Nov 02 22:52:11.794458 ip-10-0-137-4 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:52:12.226715 ip-10-0-137-4 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:52:12.226715 ip-10-0-137-4 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:52:12.226715 ip-10-0-137-4 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:52:12.226715 ip-10-0-137-4 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:52:12.232860 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232765 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:12.232860 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232857 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:12.232860 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232863 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:12.232860 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232868 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232894 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232899 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232903 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232906 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232909 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232911 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232914 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232916 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232919 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232922 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232925 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232928 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232931 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232933 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232936 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232939 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232946 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232949 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:12.233040 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232952 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232955 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232957 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232960 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232963 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232965 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232969 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232972 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232974 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232977 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232979 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232982 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232985 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232988 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232990 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232994 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.232997 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233000 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233002 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233005 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233008 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:12.233518 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233010 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233013 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233016 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233018 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233021 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233023 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233026 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233030 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233033 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233036 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233038 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233041 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233044 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233047 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233050 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233053 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233055 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233058 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233061 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233064 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:12.234022 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233066 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233070 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233073 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233076 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233078 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233081 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233085 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233088 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233091 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233094 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233096 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233099 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233101 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233104 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233106 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233109 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233111 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233114 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233117 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:12.234515 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233119 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233122 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233125 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233127 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233130 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233133 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233135 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233157 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233164 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233167 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233170 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233173 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233176 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233179 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233181 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233184 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233187 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233191 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233195 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:12.234988 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233198 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233200 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233203 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233206 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.233209 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234024 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234031 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234034 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234038 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234040 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234043 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234046 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234048 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234051 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234054 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234057 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234060 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234063 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234066 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234069 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:12.235449 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234072 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234074 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234077 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234079 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234082 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234084 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234087 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234090 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234092 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234095 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234097 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234101 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234104 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234107 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234110 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234113 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234115 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234118 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234121 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:12.235958 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234123 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234126 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234128 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234131 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234134 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234136 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234154 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234156 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234159 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234163 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234166 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234168 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234171 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234174 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234176 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234179 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234181 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234184 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234187 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234190 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:12.236445 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234192 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234195 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234197 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234200 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234204 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234207 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234209 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234212 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234215 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234218 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234221 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234224 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234227 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234230 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234232 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234235 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234238 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234240 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234243 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:12.236957 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234245 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234248 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234253 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234257 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234260 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234263 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234265 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234268 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234271 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234273 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234276 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234279 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234282 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234284 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234287 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234296 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234302 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234305 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234309 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:12.237462 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234312 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234315 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234319 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234322 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234324 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234327 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234330 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234332 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234335 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234338 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234340 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234343 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234346 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234348 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234852 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234857 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234866 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234869 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234873 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234876 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:12.237919 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234878 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234881 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234884 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234887 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234889 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234892 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234895 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234897 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234900 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234902 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234905 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234908 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234911 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234913 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234916 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234918 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234921 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234923 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234926 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234928 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:12.238428 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234931 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234934 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234936 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234939 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234942 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234945 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234947 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234950 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234953 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234959 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234962 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234964 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234967 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234969 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234972 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234975 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234977 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234980 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234982 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234985 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:12.238898 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234987 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234990 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234993 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234995 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.234998 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235001 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235004 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235006 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235009 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235011 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235015 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235017 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235020 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235022 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235025 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235027 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235030 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235034 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235037 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:12.239372 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235039 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235042 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235044 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235048 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235050 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235053 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235055 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235059 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235061 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235064 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235066 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235069 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235071 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235074 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235077 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235081 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235085 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235089 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235092 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:12.239832 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235095 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235098 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235100 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235103 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235106 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235109 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235111 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235114 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235116 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235119 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235121 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235124 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235126 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235129 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235131 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235134 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235149 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235158 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235161 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235163 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:12.240287 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235166 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.235171 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236169 2574 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236179 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236190 2574 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236196 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236201 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236204 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236208 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236213 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236216 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236219 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236222 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236226 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236229 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236232 2574 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236235 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236239 2574 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236241 2574 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236245 2574 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236251 2574 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236254 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236257 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:52:12.240773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236260 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236263 2574 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236268 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236271 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236274 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236277 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236280 2574 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236290 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236293 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236297 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236304 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236312 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236315 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236318 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236321 2574 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236324 2574 flags.go:64] FLAG: --enable-server="true" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236328 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236335 2574 flags.go:64] FLAG: --event-burst="100" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236339 2574 flags.go:64] FLAG: --event-qps="50" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236342 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236345 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236348 2574 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236352 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236355 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236359 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:52:12.241331 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236362 2574 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236366 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236369 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236372 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236374 2574 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236378 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236381 2574 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236383 2574 flags.go:64] FLAG: --feature-gates="" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236388 2574 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236391 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236394 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236398 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236401 2574 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236404 2574 flags.go:64] FLAG: --help="false" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236407 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236414 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236417 2574 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236421 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236424 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236428 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236431 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236434 2574 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236437 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:52:12.241979 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236440 2574 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236443 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236447 2574 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236451 2574 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236454 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236458 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236461 2574 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236464 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236467 2574 flags.go:64] FLAG: --lock-file="" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236474 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236477 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236480 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236486 2574 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236489 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236493 2574 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236496 2574 flags.go:64] FLAG: --logging-format="text" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236499 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236502 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236505 2574 flags.go:64] FLAG: --manifest-url="" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236508 2574 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236513 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236517 2574 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236521 2574 flags.go:64] FLAG: --max-pods="110" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236524 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236527 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:52:12.242547 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236530 2574 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236534 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236537 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236540 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236543 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236550 2574 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236554 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236557 2574 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236560 2574 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236566 2574 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236570 2574 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236573 2574 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236576 2574 flags.go:64] FLAG: --port="10250" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236579 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236582 2574 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c69704dfa6bc1438" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236586 2574 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236590 2574 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236593 2574 flags.go:64] FLAG: --register-node="true" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236597 2574 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236600 2574 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236603 2574 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236606 2574 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236609 2574 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236613 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236617 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236620 2574 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:52:12.243177 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236623 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236625 2574 flags.go:64] FLAG: --runonce="false" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236629 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236632 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236635 2574 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236638 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236641 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236645 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236648 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236652 2574 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236655 2574 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236658 2574 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236661 2574 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236664 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236667 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236671 2574 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236674 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236680 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236683 2574 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236686 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236692 2574 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236696 2574 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236700 2574 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236703 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236706 2574 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:52:12.243797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236709 2574 flags.go:64] FLAG: --v="2" Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236714 2574 flags.go:64] FLAG: --version="false" Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236719 2574 flags.go:64] FLAG: --vmodule="" Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236723 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.236726 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236864 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236868 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236871 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236874 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236877 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236880 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236883 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236886 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236888 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236891 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236894 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236897 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236900 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236902 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236905 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236908 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:12.244393 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236911 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236915 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236918 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236921 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236924 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236927 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236930 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236932 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236936 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236939 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236941 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236944 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236947 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236949 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236952 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236955 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236957 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236960 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236963 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236966 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:12.244894 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236969 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236972 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236975 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236977 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236980 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236982 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236985 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236988 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236991 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236993 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.236996 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237000 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237002 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237006 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237008 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237011 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237014 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237016 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237019 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237022 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:12.245453 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237026 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237029 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237031 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237034 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237036 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237039 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237042 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237045 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237048 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237052 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237055 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237058 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237061 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237063 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237065 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237068 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237072 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237076 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237079 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:12.245922 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237082 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237091 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237094 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237097 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237099 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237102 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237106 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237109 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237112 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237114 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237117 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237119 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237122 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237126 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237129 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237132 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237161 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237167 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237171 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:12.246427 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237174 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237176 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237179 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237182 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237184 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237187 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237190 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237193 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237195 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237198 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237200 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237203 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.237231 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237270 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237275 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237279 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:12.246883 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237282 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237284 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237287 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237290 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237294 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237297 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237299 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237302 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237305 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237307 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237312 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237315 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237317 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237320 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237323 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237325 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237328 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237330 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237333 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237336 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:12.247283 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237339 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237341 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237344 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237346 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237349 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237351 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237354 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237357 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237360 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237363 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237365 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237368 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237371 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237373 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237376 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237379 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237383 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237386 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237388 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237391 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:12.247753 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237394 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237397 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237401 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237404 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237406 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237409 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237412 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237414 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237417 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237420 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237423 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237426 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237428 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237431 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237433 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237436 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237440 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237444 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237447 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237451 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:12.248229 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237454 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237457 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237460 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237463 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237467 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237471 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237474 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237477 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237480 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237483 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237485 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237488 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237490 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237493 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237497 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237499 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237502 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237504 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:12.248694 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237507 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237510 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237512 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237515 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237518 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237520 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237523 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237525 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237528 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237530 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237533 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237536 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237539 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237541 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237544 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237547 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237549 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237552 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237555 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:12.249122 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237557 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237560 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237563 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237566 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237568 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.237571 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.244750 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.244771 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.245852 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.245858 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245920 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245927 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245930 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245934 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245938 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245941 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245944 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245947 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245950 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:12.249651 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245953 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245955 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245958 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245961 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245964 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245967 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245969 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245972 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245975 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245977 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245982 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245985 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245989 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245992 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245995 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.245998 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246000 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246003 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246006 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:12.250167 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246008 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246013 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246016 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246018 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246021 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246024 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246027 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246029 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246032 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246035 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246038 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246040 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246043 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246046 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246048 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246051 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246054 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246058 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246062 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:12.250658 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246065 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246068 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246071 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246074 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246076 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246079 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246082 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246084 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246087 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246090 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246092 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246095 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246097 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246100 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246103 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246107 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246109 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246112 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246115 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246117 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:12.251116 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246120 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246123 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246126 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246128 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246131 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246133 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246136 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246152 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246155 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246157 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246160 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246163 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246166 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246168 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246171 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246174 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246176 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246179 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246182 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246185 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:12.251630 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246188 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246190 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246193 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246195 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246198 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246201 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246204 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246207 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246211 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246213 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246216 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246219 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246221 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246224 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246227 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246229 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246232 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246234 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:12.252105 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246237 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.246266 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246470 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246475 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246478 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246481 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246483 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246486 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246488 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246491 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246494 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246496 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246499 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246502 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246504 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246507 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:12.253343 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246510 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246513 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246515 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246518 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246521 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246523 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246526 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246534 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246537 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246539 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246542 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246545 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246547 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246550 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246552 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246555 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246558 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246561 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246564 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:12.253739 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246566 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246569 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246571 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246574 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246577 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246579 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246582 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246585 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246587 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246590 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246593 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246595 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246598 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246600 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246603 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246605 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246608 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246610 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246613 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246615 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:12.254264 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246618 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246626 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246629 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246632 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246634 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246638 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246642 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246645 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246648 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246650 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246653 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246655 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246658 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246660 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246663 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246666 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246668 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246670 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246673 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246676 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:12.254804 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246678 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246681 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246685 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246689 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246691 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246694 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246697 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246699 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246702 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246705 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246707 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246714 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246717 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246720 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246728 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246731 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246734 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246736 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246739 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:12.255529 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246742 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246745 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246747 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246750 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246753 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246755 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246758 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246760 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246763 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246765 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246768 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246770 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246773 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246775 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.246800 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246837 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:52:12.255960 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246841 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246844 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246847 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246849 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246852 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246855 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246858 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246860 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246863 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246867 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246870 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246874 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246882 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246885 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246888 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246891 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246894 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246897 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246899 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:52:12.256355 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246902 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246905 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246907 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246910 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246913 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246915 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246918 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246922 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246926 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246929 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246931 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246934 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246937 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246939 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246942 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246944 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246947 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246949 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246952 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:52:12.256822 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246955 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246957 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246960 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246962 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246965 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246968 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246970 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246981 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246984 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246987 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246989 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246992 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246995 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.246997 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247000 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247002 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247005 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247007 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247010 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247012 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:52:12.257278 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247015 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247017 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247020 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247022 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247025 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247027 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247031 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247033 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247036 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247038 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247041 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247043 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247046 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247049 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247051 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247054 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247057 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247059 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247061 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247066 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:52:12.257783 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247074 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247077 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247080 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247082 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247085 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247088 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247090 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247093 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247095 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247098 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247101 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247103 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247106 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247108 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247111 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247114 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247116 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247119 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247122 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247125 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:52:12.258322 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247127 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247130 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247132 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247135 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247151 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247156 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:12.247159 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.247224 2574 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.249683 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.250753 2574 server.go:1014] "Starting client certificate rotation" Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.250871 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.251656 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:52:12.258869 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.252768 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:52:12.259260 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.259193 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:52:12.271109 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.271092 2574 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:52:12.275924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.275909 2574 log.go:25] "Validated CRI v1 image API" Nov 02 22:52:12.276780 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.276749 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:52:12.281746 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.281722 2574 fs.go:135] Filesystem UUIDs: map[2604c32e-02e9-4bee-9f55-1a62368e91c4:/dev/nvme0n1p4 5bcfb583-c16d-44bd-a8e8-ed341fc3c4ef:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 02 22:52:12.281860 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.281743 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 22:52:12.287024 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.286903 2574 manager.go:217] Machine: {Timestamp:2026-11-02 22:52:12.285591531 +0000 UTC m=+0.357028640 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100058 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a6ce6cbc56d06a867715f9b0a6994 SystemUUID:ec2a6ce6-cbc5-6d06-a867-715f9b0a6994 BootID:13a6952a-1544-4755-bfdc-8dc93267713e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f6:44:2f:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f6:44:2f:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:36:34:0e:a1:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:52:12.287024 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.287018 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:52:12.287200 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.287132 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:52:12.290513 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.290480 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:52:12.290695 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.290514 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-4.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:52:12.290749 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.290700 2574 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:52:12.290749 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.290711 2574 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:52:12.290804 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.290752 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:52:12.290804 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.290770 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:52:12.291642 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.291630 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:52:12.296984 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.296968 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:52:12.297169 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.297158 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:52:12.298005 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.297994 2574 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:52:12.298040 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.298012 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:52:12.298040 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.298030 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:52:12.298151 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.298045 2574 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:52:12.298151 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.298056 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:52:12.298534 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.298506 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:52:12.301358 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.301343 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:52:12.306478 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.306464 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:52:12.307257 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.307244 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:52:12.307296 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.307276 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:52:12.307439 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.307427 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:52:12.307474 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.307449 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:52:12.307474 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.307459 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:52:12.307474 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.307469 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:52:12.307563 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.307477 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:52:12.307563 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.307486 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:52:12.308054 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.308042 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:52:12.308086 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.308057 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:52:12.308086 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.308069 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:52:12.308086 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.308079 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:52:12.308175 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.308101 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:52:12.308175 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.308115 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:52:12.308862 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.308850 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:52:12.308898 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.308865 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:52:12.314516 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.314495 2574 server.go:1263] "Started kubelet" Nov 02 22:52:12.314620 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.314559 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:52:12.314673 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.314636 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:52:12.314721 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.314681 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:52:12.315234 ip-10-0-137-4 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:52:12.316797 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.316783 2574 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:52:12.318745 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.318728 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:52:12.320414 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.320395 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 22:52:12.323311 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.323114 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:52:12.323311 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.323125 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:52:12.323311 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.323196 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:12.323954 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.323816 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:52:12.323954 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.323818 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:52:12.323954 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.323840 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:52:12.323954 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.323921 2574 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:52:12.323954 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.323930 2574 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:52:12.328568 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.328549 2574 factory.go:153] Registering CRI-O factory Nov 02 22:52:12.328676 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.328572 2574 factory.go:223] Registration of the crio container factory successfully Nov 02 22:52:12.328676 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.328636 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:52:12.328676 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.328647 2574 factory.go:55] Registering systemd factory Nov 02 22:52:12.328676 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.328656 2574 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:52:12.328676 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.328672 2574 factory.go:103] Registering Raw factory Nov 02 22:52:12.328874 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.328687 2574 manager.go:1199] Started watching for new ooms in manager Nov 02 22:52:12.329719 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.329702 2574 manager.go:319] Starting recovery of all containers Nov 02 22:52:12.340494 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.339722 2574 manager.go:324] Recovery completed Nov 02 22:52:12.345680 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.345664 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:12.347697 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.347669 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:12.347783 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.347713 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:12.347783 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.347727 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:12.348352 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.348333 2574 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:52:12.348352 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.348348 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:12.348500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.348350 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:52:12.348500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.348382 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:52:12.349003 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.348988 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:12.349067 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.349016 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:12.349067 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.349030 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:12.349933 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.349922 2574 policy_none.go:50] "Start" Nov 02 22:52:12.349987 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.349937 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:52:12.349987 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.349950 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:52:12.351150 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.351131 2574 policy_none.go:44] "Start" Nov 02 22:52:12.404758 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.404716 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:52:12.407193 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.407181 2574 manager.go:341] "Starting Device Plugin manager" Nov 02 22:52:12.407261 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.407208 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:52:12.407261 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.407216 2574 server.go:87] "Starting device plugin registration server" Nov 02 22:52:12.407456 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.407444 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:52:12.407525 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.407459 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:52:12.407921 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.407607 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:52:12.407921 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.407716 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:52:12.407921 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.407724 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:52:12.408109 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.408092 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:52:12.408185 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.408165 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:12.473563 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.445754 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-4.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 22:52:12.473563 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.445750 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-4.us-west-2.compute.internal.18e45c3489745ead default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-4.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-4.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:52:12.314459821 +0000 UTC m=+0.385896930,LastTimestamp:2026-11-02 22:52:12.314459821 +0000 UTC m=+0.385896930,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-4.us-west-2.compute.internal,}" Nov 02 22:52:12.490013 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.489950 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-4.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 22:52:12.498943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.498914 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:52:12.500130 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.500114 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:52:12.500229 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.500154 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:52:12.500229 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.500178 2574 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:52:12.500369 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.500354 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 22:52:12.507868 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.507849 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:12.508682 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.508668 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:12.508773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.508700 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:12.508773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.508715 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:12.508773 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.508747 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.510128 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.510054 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-4.us-west-2.compute.internal.18e45c348b6f83a7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-4.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-4.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-4.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:52:12.347696039 +0000 UTC m=+0.419133157,LastTimestamp:2026-11-02 22:52:12.347696039 +0000 UTC m=+0.419133157,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-4.us-west-2.compute.internal,}" Nov 02 22:52:12.584126 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.584093 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:52:12.584295 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.584152 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gpbkx" Nov 02 22:52:12.592220 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.592188 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 22:52:12.592382 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.592247 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-4.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 22:52:12.592382 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.592186 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-4.us-west-2.compute.internal.18e45c348b6fded8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-4.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-4.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-4.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:52:12.347719384 +0000 UTC m=+0.419156492,LastTimestamp:2026-11-02 22:52:12.347719384 +0000 UTC m=+0.419156492,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-4.us-west-2.compute.internal,}" Nov 02 22:52:12.592772 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.592308 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 22:52:12.600845 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.600815 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal"] Nov 02 22:52:12.600913 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.600885 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:12.601798 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.601782 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:12.601866 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.601812 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:12.601866 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.601828 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:12.603298 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.603285 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:12.603442 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.603429 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.603483 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.603462 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:12.604052 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.604036 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:12.604168 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.604037 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:12.604168 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.604089 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:12.604168 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.604101 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:12.604168 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.604067 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:12.604168 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.604156 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:12.605410 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.605394 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.605478 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.605423 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:12.606198 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.606184 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:12.606273 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.606210 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:12.606273 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.606224 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:12.625219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.625194 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e49e94ef0b64e567c61a7cf70cf3f00-config\") pod \"kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"4e49e94ef0b64e567c61a7cf70cf3f00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.625343 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.625228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.625343 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.625259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.626060 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.625925 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" node="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.629154 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.629125 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" node="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.658470 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.658445 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:52:12.658701 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.658686 2574 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-137-4.us-west-2.compute.internal\" not found" node="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.726227 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.726198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.726401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.726231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.726401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.726255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e49e94ef0b64e567c61a7cf70cf3f00-config\") pod \"kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"4e49e94ef0b64e567c61a7cf70cf3f00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.726401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.726285 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.726401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.726335 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.726401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.726352 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e49e94ef0b64e567c61a7cf70cf3f00-config\") pod \"kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"4e49e94ef0b64e567c61a7cf70cf3f00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.753886 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.753819 2574 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-4.us-west-2.compute.internal\" not found" node="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.859025 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.858991 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:12.860182 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.860159 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:12.860286 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.860197 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:12.860286 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.860207 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:12.860286 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.860244 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.912627 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:12.912610 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:12.912677 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:12.912631 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-4.us-west-2.compute.internal\": node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:14.213698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:14.213682 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:14.216489 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:14.216469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:14.308580 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:14.308518 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:14.409204 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:14.409159 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:14.510012 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:14.509969 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:14.534134 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:14.534110 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 22:52:14.534620 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:14.534280 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:52:14.534620 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:14.534260 2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a2b73d5d16d574ed99182329d58f278a-a10e0d37d7b270df.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-4.us-west-2.compute.internal.18e45c348b6fded8\": read tcp 10.0.137.4:44686->52.14.154.205:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-4.us-west-2.compute.internal.18e45c348b6fded8 default 5497 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-4.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-4.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-4.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:52:12 +0000 UTC,LastTimestamp:2026-11-02 22:52:12.601819233 +0000 UTC m=+0.673256343,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-4.us-west-2.compute.internal,}" Nov 02 22:52:14.610930 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:14.610870 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:14.711500 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:14.711472 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:14.812505 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:14.812477 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:14.872512 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:14.872419 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:47:13 +0000 UTC" deadline="2028-06-08 06:54:05.589618544 +0000 UTC" Nov 02 22:52:14.872512 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:14.872481 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14000h1m50.717141328s" Nov 02 22:52:14.912910 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:14.912883 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:14.922024 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:14.922003 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:52:14.923069 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:14.923049 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:52:14.923181 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:14.923081 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:52:14.923181 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:14.923092 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:52:15.013880 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.013854 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:15.114006 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.113984 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:15.160735 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.160676 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:52:15.215046 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.215021 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:15.315776 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.315748 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:15.387612 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:15.387570 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8b21576a61098f70fabb74f44c4beb.slice/crio-4d634617a9a6f7d958a1241de51b5b4acedff197bdbe5fd0acf84bab0000d5d5 WatchSource:0}: Error finding container 4d634617a9a6f7d958a1241de51b5b4acedff197bdbe5fd0acf84bab0000d5d5: Status 404 returned error can't find the container with id 4d634617a9a6f7d958a1241de51b5b4acedff197bdbe5fd0acf84bab0000d5d5 Nov 02 22:52:15.387860 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:15.387845 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e49e94ef0b64e567c61a7cf70cf3f00.slice/crio-5d3263b7b47ac08bcd8d88fe8109bf9403ecf5ca6adc100a11549c9a346ed4dc WatchSource:0}: Error finding container 5d3263b7b47ac08bcd8d88fe8109bf9403ecf5ca6adc100a11549c9a346ed4dc: Status 404 returned error can't find the container with id 5d3263b7b47ac08bcd8d88fe8109bf9403ecf5ca6adc100a11549c9a346ed4dc Nov 02 22:52:15.393048 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.393031 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:52:15.416792 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.416746 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:52:15.437337 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.437314 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:52:15.507529 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.507500 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:15.512922 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.512906 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:52:15.557857 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.557839 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:52:15.561728 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.561714 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:52:15.562765 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.562753 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:52:15.582861 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.582846 2574 apiserver.go:52] "Watching apiserver" Nov 02 22:52:15.608045 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.608029 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 22:52:15.617459 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.617443 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:52:15.678698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.678647 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:52:15.678931 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.678913 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-2wfzr","openshift-image-registry/node-ca-p6x2b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-cqbm6","openshift-multus/multus-bfsqm","openshift-cluster-node-tuning-operator/tuned-527vp","openshift-dns/node-resolver-rbhdf","openshift-multus/network-metrics-daemon-zkbf8","openshift-network-operator/iptables-alerter-xv5ps","kube-system/konnectivity-agent-m2smd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc","openshift-ovn-kubernetes/ovnkube-node-pwwbw","kube-system/global-pull-secret-syncer-9b6c4"] Nov 02 22:52:15.681410 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.681393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:52:15.683116 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.683102 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.684492 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.684476 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:52:15.686810 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.686783 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:15.686906 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.686844 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:15.688594 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.688569 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.690389 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.690371 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.691746 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.691732 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:15.691815 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.691779 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:15.693214 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.693197 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.695456 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.695438 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:15.695543 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.695498 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:15.696784 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.696765 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:52:15.698231 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.698213 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:15.700144 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.700126 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.708479 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.708464 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:52:15.721441 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-run\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.721534 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721457 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-sys\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.721534 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtsgl\" (UniqueName: \"kubernetes.io/projected/526ff9d4-f30b-4414-849b-c3e747329dd9-kube-api-access-xtsgl\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.721534 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721519 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-registration-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.721654 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-host-slash\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:52:15.721654 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721565 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-daemon-config\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.721654 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.721654 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-etc-selinux\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.721815 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721668 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-netns\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.721815 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-netd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.721815 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cnibin\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.721815 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-tuned\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.721815 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-system-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.721815 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721766 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-multus\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.721815 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtz2w\" (UniqueName: \"kubernetes.io/projected/ae18bf8e-ca0b-41e8-90ca-9e830160012a-kube-api-access-jtz2w\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:52:15.722064 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-ovn\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.722064 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-config\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.722064 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-systemd\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.722064 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-bin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.722064 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-log-socket\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.722064 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.722064 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkvpl\" (UniqueName: \"kubernetes.io/projected/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-kube-api-access-gkvpl\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:52:15.722064 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721980 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-conf-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.722064 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.721999 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:15.722064 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722025 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-var-lib-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.722064 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722050 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrnjr\" (UniqueName: \"kubernetes.io/projected/0798910d-3562-4dfe-b069-587510b73467-kube-api-access-qrnjr\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722079 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysconfig\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-agent-certs\") pod \"konnectivity-agent-m2smd\" (UID: \"52e9c219-9e54-43c4-9dae-7d60c8e33dfe\") " pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722118 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-k8s-cni-cncf-io\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722141 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlv6j\" (UniqueName: \"kubernetes.io/projected/889d0ebb-164e-4c18-973a-509f4dcfa1bc-kube-api-access-tlv6j\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-device-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cni-binary-copy\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722272 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-socket-dir-parent\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-kubelet\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovn-node-metrics-cert\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722360 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-host\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-sys-fs\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722443 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-etc-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-script-lib\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.722500 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722493 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kbgs\" (UniqueName: \"kubernetes.io/projected/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-kube-api-access-9kbgs\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722509 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722554 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722595 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-iptables-alerter-script\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722629 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-kubelet\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722655 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-serviceca\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722691 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae18bf8e-ca0b-41e8-90ca-9e830160012a-hosts-file\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae18bf8e-ca0b-41e8-90ca-9e830160012a-tmp-dir\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-bin\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722777 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-lib-modules\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722860 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-tmp\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-konnectivity-ca\") pod \"konnectivity-agent-m2smd\" (UID: \"52e9c219-9e54-43c4-9dae-7d60c8e33dfe\") " pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722926 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cnibin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.722943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-netns\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.722998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723030 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-var-lib-kubelet\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-socket-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723129 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-kubelet-config\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-hostroot\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-host\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723234 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-multus-certs\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-systemd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723291 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-node-log\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-conf\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723345 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97g84\" (UniqueName: \"kubernetes.io/projected/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kube-api-access-97g84\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.723401 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723390 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkstf\" (UniqueName: \"kubernetes.io/projected/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-kube-api-access-jkstf\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:52:15.723929 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:15.723929 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723471 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-etc-kubernetes\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.723929 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-systemd-units\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.723929 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723529 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-env-overrides\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.723929 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723557 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-system-cni-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.723929 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcmgd\" (UniqueName: \"kubernetes.io/projected/c8ae4d29-9676-4fff-a0cb-a26179a61e50-kube-api-access-hcmgd\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.723929 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-kubernetes\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.723929 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723618 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-dbus\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:15.723929 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-slash\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.723929 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.723670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-modprobe-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.730229 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.730210 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:15.730229 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.730210 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:15.730373 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.730363 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:52:15.730426 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.730377 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77brg\"" Nov 02 22:52:15.731618 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.731598 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:52:15.731687 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.731638 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:52:15.731687 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.731665 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:52:15.731765 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.731684 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:52:15.731765 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.731699 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:52:15.731765 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.731734 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c9lzr\"" Nov 02 22:52:15.731856 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.731830 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:52:15.732551 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.732535 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qgnp\"" Nov 02 22:52:15.759214 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.759187 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:52:15.759214 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.759195 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9p98l\"" Nov 02 22:52:15.759469 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.759448 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-92nqq\"" Nov 02 22:52:15.759512 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.759473 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rxqrv\"" Nov 02 22:52:15.759512 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.759476 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v5hvl\"" Nov 02 22:52:15.774910 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.774893 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:52:15.776364 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776349 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:52:15.776428 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776376 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:52:15.776489 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776470 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sdz5j\"" Nov 02 22:52:15.776598 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776587 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:52:15.776685 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776606 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:52:15.776738 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776722 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:52:15.776804 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776763 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:52:15.776804 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776773 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:52:15.776939 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776882 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:52:15.776995 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776939 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:52:15.776995 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776957 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:52:15.777096 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.776987 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:52:15.777649 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.777633 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:52:15.777730 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.777699 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:52:15.777784 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.777753 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:52:15.777831 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.777813 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k5fwx" Nov 02 22:52:15.777872 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.777864 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:52:15.777915 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.777885 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:52:15.777961 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.777941 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-crtkq\"" Nov 02 22:52:15.778101 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.778084 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:52:15.788290 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.788255 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" event={"ID":"fb8b21576a61098f70fabb74f44c4beb","Type":"ContainerStarted","Data":"4d634617a9a6f7d958a1241de51b5b4acedff197bdbe5fd0acf84bab0000d5d5"} Nov 02 22:52:15.789240 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.789219 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" event={"ID":"4e49e94ef0b64e567c61a7cf70cf3f00","Type":"ContainerStarted","Data":"5d3263b7b47ac08bcd8d88fe8109bf9403ecf5ca6adc100a11549c9a346ed4dc"} Nov 02 22:52:15.824263 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-conf\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.824355 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97g84\" (UniqueName: \"kubernetes.io/projected/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kube-api-access-97g84\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.824355 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824295 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkstf\" (UniqueName: \"kubernetes.io/projected/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-kube-api-access-jkstf\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:52:15.824355 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:15.824355 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-etc-kubernetes\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824360 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-systemd-units\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-env-overrides\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824410 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-system-cni-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.824427 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcmgd\" (UniqueName: \"kubernetes.io/projected/c8ae4d29-9676-4fff-a0cb-a26179a61e50-kube-api-access-hcmgd\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-kubernetes\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824470 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-etc-kubernetes\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824432 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-conf\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824465 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-systemd-units\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.824489 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret podName:ad49b548-e79b-43aa-9479-57c1e6d06fdd nodeName:}" failed. No retries permitted until 2026-11-02 22:52:16.324469663 +0000 UTC m=+3.111898621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret") pod "global-pull-secret-syncer-9b6c4" (UID: "ad49b548-e79b-43aa-9479-57c1e6d06fdd") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824525 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-kubernetes\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824532 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-system-cni-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.824582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-dbus\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-slash\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-modprobe-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824661 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-run\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824680 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-dbus\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824683 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-slash\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-sys\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824733 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-sys\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtsgl\" (UniqueName: \"kubernetes.io/projected/526ff9d4-f30b-4414-849b-c3e747329dd9-kube-api-access-xtsgl\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824747 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-modprobe-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824746 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-run\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-registration-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-host-slash\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824814 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-daemon-config\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824841 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-host-slash\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824860 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-env-overrides\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.825222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-etc-selinux\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824830 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-registration-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-netns\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-netd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cnibin\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824981 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-etc-selinux\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.824986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-tuned\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825003 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-netd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825017 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-system-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-multus\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtz2w\" (UniqueName: \"kubernetes.io/projected/ae18bf8e-ca0b-41e8-90ca-9e830160012a-kube-api-access-jtz2w\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825076 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-system-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825014 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cnibin\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825042 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-netns\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825073 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-multus\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-ovn\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-config\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-systemd\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.825924 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825186 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-ovn\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825199 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-bin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-log-socket\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825266 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-bin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825280 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-systemd\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825300 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkvpl\" (UniqueName: \"kubernetes.io/projected/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-kube-api-access-gkvpl\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825344 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-conf-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825352 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825320 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-log-socket\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825405 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-conf-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-var-lib-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825436 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-daemon-config\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825429 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrnjr\" (UniqueName: \"kubernetes.io/projected/0798910d-3562-4dfe-b069-587510b73467-kube-api-access-qrnjr\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825495 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-var-lib-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.826698 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysconfig\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-agent-certs\") pod \"konnectivity-agent-m2smd\" (UID: \"52e9c219-9e54-43c4-9dae-7d60c8e33dfe\") " pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825544 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysconfig\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-k8s-cni-cncf-io\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlv6j\" (UniqueName: \"kubernetes.io/projected/889d0ebb-164e-4c18-973a-509f4dcfa1bc-kube-api-access-tlv6j\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-device-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cni-binary-copy\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-socket-dir-parent\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825667 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-kubelet\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825672 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-device-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825678 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-k8s-cni-cncf-io\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovn-node-metrics-cert\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-host\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825717 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825724 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-kubelet\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825728 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-socket-dir-parent\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-sys-fs\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.827542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825767 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-sys-fs\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-etc-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825772 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-host\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-script-lib\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825793 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-etc-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kbgs\" (UniqueName: \"kubernetes.io/projected/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-kube-api-access-9kbgs\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-iptables-alerter-script\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-kubelet\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-serviceca\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae18bf8e-ca0b-41e8-90ca-9e830160012a-hosts-file\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae18bf8e-ca0b-41e8-90ca-9e830160012a-tmp-dir\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825981 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-bin\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.825996 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-lib-modules\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.828219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-tmp\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-konnectivity-ca\") pod \"konnectivity-agent-m2smd\" (UID: \"52e9c219-9e54-43c4-9dae-7d60c8e33dfe\") " pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cnibin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826072 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-netns\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-var-lib-kubelet\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-socket-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826216 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-kubelet-config\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826239 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-hostroot\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-host\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-multus-certs\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-systemd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826322 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-config\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-node-log\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.828667 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826364 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826380 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-node-log\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826440 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826555 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826624 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-var-lib-kubelet\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826705 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-script-lib\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826748 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-socket-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826778 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae18bf8e-ca0b-41e8-90ca-9e830160012a-hosts-file\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826794 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826842 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826860 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cni-binary-copy\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826903 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-konnectivity-ca\") pod \"konnectivity-agent-m2smd\" (UID: \"52e9c219-9e54-43c4-9dae-7d60c8e33dfe\") " pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826901 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826929 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-kubelet\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826971 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-kubelet-config\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826978 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-multus-certs\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.826986 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-bin\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.827018 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-systemd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.829310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.827032 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-host\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.827068 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-hostroot\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.827236 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cnibin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.827423 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-iptables-alerter-script\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.827496 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-serviceca\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.827597 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae18bf8e-ca0b-41e8-90ca-9e830160012a-tmp-dir\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.827680 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.827726 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs podName:0798910d-3562-4dfe-b069-587510b73467 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:16.327711034 +0000 UTC m=+3.115139991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs") pod "network-metrics-daemon-zkbf8" (UID: "0798910d-3562-4dfe-b069-587510b73467") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.827773 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-netns\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.827861 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-lib-modules\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.827922 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.828093 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.828858 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-tuned\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.829024 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovn-node-metrics-cert\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.829033 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-agent-certs\") pod \"konnectivity-agent-m2smd\" (UID: \"52e9c219-9e54-43c4-9dae-7d60c8e33dfe\") " pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:15.829774 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.829165 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-tmp\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.877578 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.877558 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:52:15.877675 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.877585 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:52:15.877675 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.877598 2574 projected.go:196] Error preparing data for projected volume kube-api-access-7pcfv for pod openshift-network-diagnostics/network-check-target-2wfzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:15.877675 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:15.877657 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv podName:b37eb397-e28e-49c2-a711-a4efd3466679 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:16.377642106 +0000 UTC m=+3.165071082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcfv" (UniqueName: "kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv") pod "network-check-target-2wfzr" (UID: "b37eb397-e28e-49c2-a711-a4efd3466679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:15.878657 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.878597 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkvpl\" (UniqueName: \"kubernetes.io/projected/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-kube-api-access-gkvpl\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:52:15.879109 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.879089 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcmgd\" (UniqueName: \"kubernetes.io/projected/c8ae4d29-9676-4fff-a0cb-a26179a61e50-kube-api-access-hcmgd\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:15.879330 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.879301 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtz2w\" (UniqueName: \"kubernetes.io/projected/ae18bf8e-ca0b-41e8-90ca-9e830160012a-kube-api-access-jtz2w\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:52:15.879512 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.879492 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtsgl\" (UniqueName: \"kubernetes.io/projected/526ff9d4-f30b-4414-849b-c3e747329dd9-kube-api-access-xtsgl\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:15.879620 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.879603 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97g84\" (UniqueName: \"kubernetes.io/projected/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kube-api-access-97g84\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:15.879815 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.879794 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkstf\" (UniqueName: \"kubernetes.io/projected/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-kube-api-access-jkstf\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:52:15.880131 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.880112 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlv6j\" (UniqueName: \"kubernetes.io/projected/889d0ebb-164e-4c18-973a-509f4dcfa1bc-kube-api-access-tlv6j\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:52:15.880391 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.880374 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrnjr\" (UniqueName: \"kubernetes.io/projected/0798910d-3562-4dfe-b069-587510b73467-kube-api-access-qrnjr\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:15.881272 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.881252 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kbgs\" (UniqueName: \"kubernetes.io/projected/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-kube-api-access-9kbgs\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:15.992554 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:15.992489 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:52:16.001193 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:16.001168 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546959a7_c5ae_4ea7_a9b7_c6ed4c0d0526.slice/crio-4981edabeec8a7f8151ba294e37df74d845c9e06642559be4ee3aacd6870b985 WatchSource:0}: Error finding container 4981edabeec8a7f8151ba294e37df74d845c9e06642559be4ee3aacd6870b985: Status 404 returned error can't find the container with id 4981edabeec8a7f8151ba294e37df74d845c9e06642559be4ee3aacd6870b985 Nov 02 22:52:16.008291 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.008273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:52:16.011262 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.011247 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:16.015139 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:16.015121 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a82c3d_2c8e_4250_b6fd_09e55c4d7b78.slice/crio-a48048e59f0de866ceb5f1c9e9ea470684afa28ea21911a4b5c325d513c55509 WatchSource:0}: Error finding container a48048e59f0de866ceb5f1c9e9ea470684afa28ea21911a4b5c325d513c55509: Status 404 returned error can't find the container with id a48048e59f0de866ceb5f1c9e9ea470684afa28ea21911a4b5c325d513c55509 Nov 02 22:52:16.018247 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.018219 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:52:16.019225 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.019212 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bfsqm" Nov 02 22:52:16.019385 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:16.019370 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584a133a_3ca7_4a1b_88fc_83d98e3abc3d.slice/crio-57800bd84f651ea3ff54b9079349ff3ce771887ec6fc9cb597e466a8b424371b WatchSource:0}: Error finding container 57800bd84f651ea3ff54b9079349ff3ce771887ec6fc9cb597e466a8b424371b: Status 404 returned error can't find the container with id 57800bd84f651ea3ff54b9079349ff3ce771887ec6fc9cb597e466a8b424371b Nov 02 22:52:16.021720 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.021642 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:52:16.028458 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.028442 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:52:16.029076 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:16.029060 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ae4d29_9676_4fff_a0cb_a26179a61e50.slice/crio-4325406e33abdacdd346295a9aa0af7461ef4610bdaeaa3aae5109b28ad93555 WatchSource:0}: Error finding container 4325406e33abdacdd346295a9aa0af7461ef4610bdaeaa3aae5109b28ad93555: Status 404 returned error can't find the container with id 4325406e33abdacdd346295a9aa0af7461ef4610bdaeaa3aae5109b28ad93555 Nov 02 22:52:16.031636 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:16.031617 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526ff9d4_f30b_4414_849b_c3e747329dd9.slice/crio-db5e32ae6c96561bcc1fb7c8856eee630a02bc8bb520be1d863f8effa7c96447 WatchSource:0}: Error finding container db5e32ae6c96561bcc1fb7c8856eee630a02bc8bb520be1d863f8effa7c96447: Status 404 returned error can't find the container with id db5e32ae6c96561bcc1fb7c8856eee630a02bc8bb520be1d863f8effa7c96447 Nov 02 22:52:16.032279 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:16.032257 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889d0ebb_164e_4c18_973a_509f4dcfa1bc.slice/crio-06a3857c77677d72436a1c3f0a7956026adcef78140423cd5fb02694de51ec39 WatchSource:0}: Error finding container 06a3857c77677d72436a1c3f0a7956026adcef78140423cd5fb02694de51ec39: Status 404 returned error can't find the container with id 06a3857c77677d72436a1c3f0a7956026adcef78140423cd5fb02694de51ec39 Nov 02 22:52:16.034441 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.034429 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:16.036561 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:16.036539 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae18bf8e_ca0b_41e8_90ca_9e830160012a.slice/crio-3d5057cd8d40b089ab4e0e3c20ae840fe474f35468fcfd6149b68e80cde2d523 WatchSource:0}: Error finding container 3d5057cd8d40b089ab4e0e3c20ae840fe474f35468fcfd6149b68e80cde2d523: Status 404 returned error can't find the container with id 3d5057cd8d40b089ab4e0e3c20ae840fe474f35468fcfd6149b68e80cde2d523 Nov 02 22:52:16.041651 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.041633 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:52:16.041967 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:16.041947 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e9c219_9e54_43c4_9dae_7d60c8e33dfe.slice/crio-00f022618f89226e2a89e07fdd6cd410aee7a2e86c1d088d79f20bfdfab4c937 WatchSource:0}: Error finding container 00f022618f89226e2a89e07fdd6cd410aee7a2e86c1d088d79f20bfdfab4c937: Status 404 returned error can't find the container with id 00f022618f89226e2a89e07fdd6cd410aee7a2e86c1d088d79f20bfdfab4c937 Nov 02 22:52:16.048639 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:16.048621 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce280d50_8be5_4223_91ce_3ea1da4e7ff6.slice/crio-b1b885e08fa88c84d5b63e15a29e322c9289437ebc9ea417adeca61da9e9dc06 WatchSource:0}: Error finding container b1b885e08fa88c84d5b63e15a29e322c9289437ebc9ea417adeca61da9e9dc06: Status 404 returned error can't find the container with id b1b885e08fa88c84d5b63e15a29e322c9289437ebc9ea417adeca61da9e9dc06 Nov 02 22:52:16.330200 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.330116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:16.330200 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.330184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:16.330402 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:16.330273 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:16.330402 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:16.330338 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret podName:ad49b548-e79b-43aa-9479-57c1e6d06fdd nodeName:}" failed. No retries permitted until 2026-11-02 22:52:17.330320456 +0000 UTC m=+4.117749397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret") pod "global-pull-secret-syncer-9b6c4" (UID: "ad49b548-e79b-43aa-9479-57c1e6d06fdd") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:16.330402 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:16.330273 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:16.330402 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:16.330389 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs podName:0798910d-3562-4dfe-b069-587510b73467 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:17.330376842 +0000 UTC m=+4.117805785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs") pod "network-metrics-daemon-zkbf8" (UID: "0798910d-3562-4dfe-b069-587510b73467") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:16.430572 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.430538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:16.430741 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:16.430692 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:52:16.430741 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:16.430714 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:52:16.430741 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:16.430725 2574 projected.go:196] Error preparing data for projected volume kube-api-access-7pcfv for pod openshift-network-diagnostics/network-check-target-2wfzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:16.430878 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:16.430778 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv podName:b37eb397-e28e-49c2-a711-a4efd3466679 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:17.430760001 +0000 UTC m=+4.218188955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcfv" (UniqueName: "kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv") pod "network-check-target-2wfzr" (UID: "b37eb397-e28e-49c2-a711-a4efd3466679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:16.781871 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.781782 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:47:15 +0000 UTC" deadline="2028-04-06 01:57:31.154059565 +0000 UTC" Nov 02 22:52:16.781871 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.781821 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12483h5m14.372242621s" Nov 02 22:52:16.785070 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.785042 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:16.785205 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:16.785160 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:16.792648 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.792620 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6x2b" event={"ID":"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78","Type":"ContainerStarted","Data":"a48048e59f0de866ceb5f1c9e9ea470684afa28ea21911a4b5c325d513c55509"} Nov 02 22:52:16.793841 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.793821 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xv5ps" event={"ID":"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526","Type":"ContainerStarted","Data":"4981edabeec8a7f8151ba294e37df74d845c9e06642559be4ee3aacd6870b985"} Nov 02 22:52:16.795047 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.795023 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" event={"ID":"ce280d50-8be5-4223-91ce-3ea1da4e7ff6","Type":"ContainerStarted","Data":"b1b885e08fa88c84d5b63e15a29e322c9289437ebc9ea417adeca61da9e9dc06"} Nov 02 22:52:16.796335 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.796309 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bfsqm" event={"ID":"889d0ebb-164e-4c18-973a-509f4dcfa1bc","Type":"ContainerStarted","Data":"06a3857c77677d72436a1c3f0a7956026adcef78140423cd5fb02694de51ec39"} Nov 02 22:52:16.797538 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.797515 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m2smd" event={"ID":"52e9c219-9e54-43c4-9dae-7d60c8e33dfe","Type":"ContainerStarted","Data":"00f022618f89226e2a89e07fdd6cd410aee7a2e86c1d088d79f20bfdfab4c937"} Nov 02 22:52:16.798909 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.798854 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rbhdf" event={"ID":"ae18bf8e-ca0b-41e8-90ca-9e830160012a","Type":"ContainerStarted","Data":"3d5057cd8d40b089ab4e0e3c20ae840fe474f35468fcfd6149b68e80cde2d523"} Nov 02 22:52:16.800207 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.800185 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-527vp" event={"ID":"526ff9d4-f30b-4414-849b-c3e747329dd9","Type":"ContainerStarted","Data":"db5e32ae6c96561bcc1fb7c8856eee630a02bc8bb520be1d863f8effa7c96447"} Nov 02 22:52:16.802238 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.802215 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" event={"ID":"c8ae4d29-9676-4fff-a0cb-a26179a61e50","Type":"ContainerStarted","Data":"4325406e33abdacdd346295a9aa0af7461ef4610bdaeaa3aae5109b28ad93555"} Nov 02 22:52:16.803517 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:16.803492 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" event={"ID":"584a133a-3ca7-4a1b-88fc-83d98e3abc3d","Type":"ContainerStarted","Data":"57800bd84f651ea3ff54b9079349ff3ce771887ec6fc9cb597e466a8b424371b"} Nov 02 22:52:17.337701 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:17.337631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:17.337882 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:17.337721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:17.337882 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:17.337782 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:17.337882 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:17.337824 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:17.337882 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:17.337853 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret podName:ad49b548-e79b-43aa-9479-57c1e6d06fdd nodeName:}" failed. No retries permitted until 2026-11-02 22:52:19.337835028 +0000 UTC m=+6.125263980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret") pod "global-pull-secret-syncer-9b6c4" (UID: "ad49b548-e79b-43aa-9479-57c1e6d06fdd") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:17.337882 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:17.337877 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs podName:0798910d-3562-4dfe-b069-587510b73467 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:19.337861685 +0000 UTC m=+6.125290644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs") pod "network-metrics-daemon-zkbf8" (UID: "0798910d-3562-4dfe-b069-587510b73467") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:17.438982 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:17.438684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:17.438982 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:17.438855 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:52:17.438982 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:17.438877 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:52:17.438982 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:17.438890 2574 projected.go:196] Error preparing data for projected volume kube-api-access-7pcfv for pod openshift-network-diagnostics/network-check-target-2wfzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:17.438982 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:17.438949 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv podName:b37eb397-e28e-49c2-a711-a4efd3466679 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:19.438930182 +0000 UTC m=+6.226359127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcfv" (UniqueName: "kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv") pod "network-check-target-2wfzr" (UID: "b37eb397-e28e-49c2-a711-a4efd3466679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:17.785591 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:17.785234 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:47:15 +0000 UTC" deadline="2028-04-29 23:25:44.462517743 +0000 UTC" Nov 02 22:52:17.785591 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:17.785269 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13056h33m26.677252919s" Nov 02 22:52:17.785591 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:17.785360 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:17.785591 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:17.785480 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:17.786426 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:17.786282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:17.786426 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:17.786387 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:18.785262 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:18.785075 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:18.785262 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:18.785218 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:19.370619 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:19.370397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:19.371075 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:19.370653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:19.371075 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:19.370801 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:19.371075 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:19.370862 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret podName:ad49b548-e79b-43aa-9479-57c1e6d06fdd nodeName:}" failed. No retries permitted until 2026-11-02 22:52:23.370844151 +0000 UTC m=+10.158273091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret") pod "global-pull-secret-syncer-9b6c4" (UID: "ad49b548-e79b-43aa-9479-57c1e6d06fdd") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:19.371075 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:19.370930 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:19.371075 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:19.370964 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs podName:0798910d-3562-4dfe-b069-587510b73467 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:23.370952323 +0000 UTC m=+10.158381263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs") pod "network-metrics-daemon-zkbf8" (UID: "0798910d-3562-4dfe-b069-587510b73467") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:19.473918 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:19.473537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:19.473918 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:19.473755 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:52:19.473918 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:19.473775 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:52:19.473918 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:19.473790 2574 projected.go:196] Error preparing data for projected volume kube-api-access-7pcfv for pod openshift-network-diagnostics/network-check-target-2wfzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:19.473918 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:19.473846 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv podName:b37eb397-e28e-49c2-a711-a4efd3466679 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:23.473828412 +0000 UTC m=+10.261257353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcfv" (UniqueName: "kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv") pod "network-check-target-2wfzr" (UID: "b37eb397-e28e-49c2-a711-a4efd3466679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:19.785488 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:19.785458 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:19.785694 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:19.785560 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:19.785763 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:19.785697 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:19.785815 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:19.785762 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:19.837654 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:19.837583 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" event={"ID":"4e49e94ef0b64e567c61a7cf70cf3f00","Type":"ContainerStarted","Data":"59f470574a254011528a975d3f9b0aedfdb7465dee9eb22c48795004e160fc8a"} Nov 02 22:52:19.846571 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:19.846529 2574 generic.go:356] "Generic (PLEG): container finished" podID="fb8b21576a61098f70fabb74f44c4beb" containerID="27c2992b77b9d0fd7a46d8e79f8cc5925194f302658f65915a10709da67c719b" exitCode=0 Nov 02 22:52:19.846696 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:19.846614 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" event={"ID":"fb8b21576a61098f70fabb74f44c4beb","Type":"ContainerDied","Data":"27c2992b77b9d0fd7a46d8e79f8cc5925194f302658f65915a10709da67c719b"} Nov 02 22:52:19.850618 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:19.850584 2574 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="707e3785a569b84beac2853f3524fa3fda6e6aeac551f68edd91bd22943715f5" exitCode=0 Nov 02 22:52:19.850716 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:19.850617 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" event={"ID":"c8ae4d29-9676-4fff-a0cb-a26179a61e50","Type":"ContainerDied","Data":"707e3785a569b84beac2853f3524fa3fda6e6aeac551f68edd91bd22943715f5"} Nov 02 22:52:20.052955 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:20.052888 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" podStartSLOduration=5.052867343 podStartE2EDuration="5.052867343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:19.942948956 +0000 UTC m=+6.730377919" watchObservedRunningTime="2026-11-02 22:52:20.052867343 +0000 UTC m=+6.840296306" Nov 02 22:52:20.785113 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:20.785072 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:20.785599 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:20.785260 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:20.862694 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:20.862644 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" event={"ID":"fb8b21576a61098f70fabb74f44c4beb","Type":"ContainerStarted","Data":"e2486101a425ee1ad164c33d3bf030ca07a60623928f5908f779740918c83ca6"} Nov 02 22:52:20.970525 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:20.970463 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" podStartSLOduration=5.970445576 podStartE2EDuration="5.970445576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:20.970110185 +0000 UTC m=+7.757539147" watchObservedRunningTime="2026-11-02 22:52:20.970445576 +0000 UTC m=+7.757874541" Nov 02 22:52:21.788258 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:21.787686 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:21.788258 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:21.787827 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:21.788816 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:21.788275 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:21.788816 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:21.788387 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:21.869300 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:21.869266 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m2smd" event={"ID":"52e9c219-9e54-43c4-9dae-7d60c8e33dfe","Type":"ContainerStarted","Data":"fc33ee04e31c20c3e92db852b5495b46ec15354cb61bd170e0016cd0c087c15d"} Nov 02 22:52:22.218563 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:22.218530 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:22.219093 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:22.219059 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:22.324834 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:22.324777 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m2smd" podStartSLOduration=3.539813562 podStartE2EDuration="8.324756244s" totalImagesPullingTime="4.784942682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:21.975995935 +0000 UTC m=+8.763424898" watchObservedRunningTime="2026-11-02 22:52:22.324756244 +0000 UTC m=+9.112185206" Nov 02 22:52:22.784644 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:22.784568 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:22.784789 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:22.784705 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:22.873244 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:22.873205 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6x2b" event={"ID":"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78","Type":"ContainerStarted","Data":"50047a1ae779a07f121fc9773122c39c70006db1043fcc524dc37d262c6b2ce3"} Nov 02 22:52:22.877416 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:22.877390 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" event={"ID":"ce280d50-8be5-4223-91ce-3ea1da4e7ff6","Type":"ContainerStarted","Data":"169db18dec03b691cdd7295abe991c5ea83fcb2d7952a59306069e3f553a4f67"} Nov 02 22:52:23.405320 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:23.405274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:23.405497 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:23.405341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:23.405497 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:23.405475 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:23.405604 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:23.405542 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret podName:ad49b548-e79b-43aa-9479-57c1e6d06fdd nodeName:}" failed. No retries permitted until 2026-11-02 22:52:31.405523335 +0000 UTC m=+18.192952276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret") pod "global-pull-secret-syncer-9b6c4" (UID: "ad49b548-e79b-43aa-9479-57c1e6d06fdd") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:23.406384 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:23.406189 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:23.406384 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:23.406241 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs podName:0798910d-3562-4dfe-b069-587510b73467 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:31.406224921 +0000 UTC m=+18.193653866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs") pod "network-metrics-daemon-zkbf8" (UID: "0798910d-3562-4dfe-b069-587510b73467") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:23.505928 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:23.505888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:23.506160 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:23.506112 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:52:23.506160 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:23.506136 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:52:23.506312 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:23.506164 2574 projected.go:196] Error preparing data for projected volume kube-api-access-7pcfv for pod openshift-network-diagnostics/network-check-target-2wfzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:23.506312 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:23.506223 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv podName:b37eb397-e28e-49c2-a711-a4efd3466679 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:31.506205056 +0000 UTC m=+18.293633997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcfv" (UniqueName: "kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv") pod "network-check-target-2wfzr" (UID: "b37eb397-e28e-49c2-a711-a4efd3466679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:23.787336 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:23.787265 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:23.787486 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:23.787407 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:23.788600 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:23.788577 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:23.788698 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:23.788668 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:23.880466 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:23.880435 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:52:24.553091 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:24.552873 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:52:24.712650 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:24.712531 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:52:24.552892647Z","UUID":"aa71c227-4be1-40dc-8f37-52d851c5b1eb","Handler":null,"Name":"","Endpoint":""} Nov 02 22:52:24.714641 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:24.714228 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:52:24.714641 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:24.714264 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:52:24.784411 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:24.784387 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:24.784529 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:24.784494 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:24.884840 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:24.884807 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" event={"ID":"c8ae4d29-9676-4fff-a0cb-a26179a61e50","Type":"ContainerStarted","Data":"5b3c1bfc04c33f4d4f65657de1661e07dbfd0b0a540c8e2d2969cc80c72bffdf"} Nov 02 22:52:24.889761 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:24.889727 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" event={"ID":"ce280d50-8be5-4223-91ce-3ea1da4e7ff6","Type":"ContainerStarted","Data":"27952e9a351a00e42e930ac87c6eb4839c63fa2cdd26178393c9d688fab48d47"} Nov 02 22:52:24.891983 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:24.891553 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bfsqm" event={"ID":"889d0ebb-164e-4c18-973a-509f4dcfa1bc","Type":"ContainerStarted","Data":"bcb3dae199436ba7c40ed1dbad9acb6089191962e9176aa5b63b532f5cfe2a8d"} Nov 02 22:52:24.995316 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:24.995260 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p6x2b" podStartSLOduration=4.769739118 podStartE2EDuration="10.99524116s" totalImagesPullingTime="6.225502042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:22.979408386 +0000 UTC m=+9.766837347" watchObservedRunningTime="2026-11-02 22:52:24.99524116 +0000 UTC m=+11.782670121" Nov 02 22:52:25.785715 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:25.785425 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:25.787983 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:25.787946 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:25.788346 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:25.788328 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:25.788438 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:25.788419 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:25.898581 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:25.898539 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rbhdf" event={"ID":"ae18bf8e-ca0b-41e8-90ca-9e830160012a","Type":"ContainerStarted","Data":"26bc6b5324f5c6c35d676ef8cf39cd47a4eb1a7bd8377b27260d68a2588cd8d9"} Nov 02 22:52:25.900881 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:25.900854 2574 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="5b3c1bfc04c33f4d4f65657de1661e07dbfd0b0a540c8e2d2969cc80c72bffdf" exitCode=0 Nov 02 22:52:25.901330 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:25.901308 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" event={"ID":"c8ae4d29-9676-4fff-a0cb-a26179a61e50","Type":"ContainerDied","Data":"5b3c1bfc04c33f4d4f65657de1661e07dbfd0b0a540c8e2d2969cc80c72bffdf"} Nov 02 22:52:26.005166 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:26.004479 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-bfsqm" podStartSLOduration=3.767823571 podStartE2EDuration="12.004461346s" totalImagesPullingTime="8.236637775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:25.102972308 +0000 UTC m=+11.890401271" watchObservedRunningTime="2026-11-02 22:52:26.004461346 +0000 UTC m=+12.791890309" Nov 02 22:52:26.005166 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:26.004669 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rbhdf" podStartSLOduration=3.173749235 podStartE2EDuration="12.004661039s" totalImagesPullingTime="8.830911804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:26.004433712 +0000 UTC m=+12.791862673" watchObservedRunningTime="2026-11-02 22:52:26.004661039 +0000 UTC m=+12.792090002" Nov 02 22:52:26.784927 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:26.784897 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:26.785074 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:26.785000 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:26.904524 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:26.904483 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xv5ps" event={"ID":"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526","Type":"ContainerStarted","Data":"3c88860c22169f3ae42bdb59f4cb7774a6f72e96665242523c2143803fb9e2d3"} Nov 02 22:52:26.906948 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:26.906917 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" event={"ID":"ce280d50-8be5-4223-91ce-3ea1da4e7ff6","Type":"ContainerStarted","Data":"a39b711264a31c2f7e4566155b1b92357115cf3eb22be221e6a13e73a5581986"} Nov 02 22:52:27.009618 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:27.009580 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xv5ps" podStartSLOduration=4.276607239 podStartE2EDuration="13.009563545s" totalImagesPullingTime="8.732956306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:27.009548262 +0000 UTC m=+13.796977226" watchObservedRunningTime="2026-11-02 22:52:27.009563545 +0000 UTC m=+13.796992498" Nov 02 22:52:27.116780 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:27.116674 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" podStartSLOduration=3.362432961 podStartE2EDuration="13.11665588s" totalImagesPullingTime="9.754222919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:27.116487573 +0000 UTC m=+13.903916537" watchObservedRunningTime="2026-11-02 22:52:27.11665588 +0000 UTC m=+13.904084843" Nov 02 22:52:27.784799 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:27.784718 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:27.784799 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:27.784762 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:27.785005 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:27.784868 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:27.785005 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:27.784983 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:27.910399 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:27.910344 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-527vp" event={"ID":"526ff9d4-f30b-4414-849b-c3e747329dd9","Type":"ContainerStarted","Data":"da3b2f455d1c6054e2435dfdaa204af208cbce343a1c3cc22f76ae48801151f3"} Nov 02 22:52:27.912792 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:27.912760 2574 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="ec742a2d51d4b5375c6d8ea1087467538e985e7cfffab8dfeb7aca00aac5206f" exitCode=0 Nov 02 22:52:27.912914 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:27.912809 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" event={"ID":"c8ae4d29-9676-4fff-a0cb-a26179a61e50","Type":"ContainerDied","Data":"ec742a2d51d4b5375c6d8ea1087467538e985e7cfffab8dfeb7aca00aac5206f"} Nov 02 22:52:28.024464 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:28.024399 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-527vp" podStartSLOduration=3.0940244789999998 podStartE2EDuration="14.024380508s" totalImagesPullingTime="10.930356029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:28.02376345 +0000 UTC m=+14.811192411" watchObservedRunningTime="2026-11-02 22:52:28.024380508 +0000 UTC m=+14.811809471" Nov 02 22:52:28.785309 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:28.785278 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:28.785466 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:28.785398 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:29.784803 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:29.784717 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:29.785419 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:29.784835 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:29.785419 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:29.784890 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:29.785419 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:29.784994 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:29.823940 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:29.823908 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:29.824085 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:29.824045 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:52:29.824731 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:29.824709 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:52:29.919732 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:29.919702 2574 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="97d0c2aec72bd37d6a2054de2c4d9d4349b0f591de7281791d2b8e0729e120f5" exitCode=0 Nov 02 22:52:29.919901 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:29.919793 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" event={"ID":"c8ae4d29-9676-4fff-a0cb-a26179a61e50","Type":"ContainerDied","Data":"97d0c2aec72bd37d6a2054de2c4d9d4349b0f591de7281791d2b8e0729e120f5"} Nov 02 22:52:30.784561 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:30.784379 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:30.784725 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:30.784626 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:31.475049 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:31.474819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:31.475486 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:31.475126 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:31.475486 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:31.475186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:31.475486 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:31.475249 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs podName:0798910d-3562-4dfe-b069-587510b73467 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:47.475228581 +0000 UTC m=+34.262657536 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs") pod "network-metrics-daemon-zkbf8" (UID: "0798910d-3562-4dfe-b069-587510b73467") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:31.475486 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:31.475301 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:31.475486 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:31.475380 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret podName:ad49b548-e79b-43aa-9479-57c1e6d06fdd nodeName:}" failed. No retries permitted until 2026-11-02 22:52:47.475362838 +0000 UTC m=+34.262791796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret") pod "global-pull-secret-syncer-9b6c4" (UID: "ad49b548-e79b-43aa-9479-57c1e6d06fdd") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:31.576210 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:31.576128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:31.576357 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:31.576303 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:52:31.576357 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:31.576330 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:52:31.576357 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:31.576343 2574 projected.go:196] Error preparing data for projected volume kube-api-access-7pcfv for pod openshift-network-diagnostics/network-check-target-2wfzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:31.576466 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:31.576408 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv podName:b37eb397-e28e-49c2-a711-a4efd3466679 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:47.576388506 +0000 UTC m=+34.363817460 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcfv" (UniqueName: "kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv") pod "network-check-target-2wfzr" (UID: "b37eb397-e28e-49c2-a711-a4efd3466679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:31.784621 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:31.784536 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:31.784745 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:31.784652 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:31.784745 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:31.784680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:31.784804 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:31.784746 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:31.927063 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:31.927020 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" event={"ID":"584a133a-3ca7-4a1b-88fc-83d98e3abc3d","Type":"ContainerStarted","Data":"3ef70e2e471014e4873fd0130630bfd9adad9cbb70a59f96aa0f8b1a4e9ae16b"} Nov 02 22:52:31.927245 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:31.927102 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" event={"ID":"584a133a-3ca7-4a1b-88fc-83d98e3abc3d","Type":"ContainerStarted","Data":"64c74053de74a3b7f01d1ec5f5b4ab22d58850b9c02d1cc722fcd1ded523abfc"} Nov 02 22:52:31.927245 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:31.927163 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" event={"ID":"584a133a-3ca7-4a1b-88fc-83d98e3abc3d","Type":"ContainerStarted","Data":"290c12c3daca6113357bb5b7b7534a273dbd6721b602597da23849cd99654416"} Nov 02 22:52:31.927245 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:31.927207 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" event={"ID":"584a133a-3ca7-4a1b-88fc-83d98e3abc3d","Type":"ContainerStarted","Data":"5a47f0d4f702689073d7db433a9209f7317bd2a0072e996defeb49826e629c68"} Nov 02 22:52:31.927382 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:31.927249 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" event={"ID":"584a133a-3ca7-4a1b-88fc-83d98e3abc3d","Type":"ContainerStarted","Data":"9a180b0a414ec8fdbb6ac12cec2ae9066d1cf39e50f1e09e05103daa997ccb4c"} Nov 02 22:52:31.927382 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:31.927299 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" event={"ID":"584a133a-3ca7-4a1b-88fc-83d98e3abc3d","Type":"ContainerStarted","Data":"39a7d646086e6fd48ef13e7a9e00a900e50a189043040aaafdbb40018cb36904"} Nov 02 22:52:32.326864 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:32.326805 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rbhdf_ae18bf8e-ca0b-41e8-90ca-9e830160012a/dns-node-resolver/0.log" Nov 02 22:52:32.785068 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:32.785000 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:32.785495 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:32.785122 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:32.931220 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:32.931186 2574 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="53f12615f0cac7c3e4705170f6bb47573e7c14965891089b2ddf984f7778d3e7" exitCode=0 Nov 02 22:52:32.931370 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:32.931239 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" event={"ID":"c8ae4d29-9676-4fff-a0cb-a26179a61e50","Type":"ContainerDied","Data":"53f12615f0cac7c3e4705170f6bb47573e7c14965891089b2ddf984f7778d3e7"} Nov 02 22:52:33.167857 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:33.167797 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p6x2b_a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78/node-ca/0.log" Nov 02 22:52:33.785018 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:33.784950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:33.785226 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:33.785027 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:33.785226 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:33.785138 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:33.785601 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:33.785274 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:33.935995 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:33.935962 2574 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="1bf23784aae8c843e6136fe839823452fdfaaf69aa4b0a0ee81e621cd2750aae" exitCode=0 Nov 02 22:52:33.936128 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:33.936042 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" event={"ID":"c8ae4d29-9676-4fff-a0cb-a26179a61e50","Type":"ContainerDied","Data":"1bf23784aae8c843e6136fe839823452fdfaaf69aa4b0a0ee81e621cd2750aae"} Nov 02 22:52:33.939317 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:33.939295 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" event={"ID":"584a133a-3ca7-4a1b-88fc-83d98e3abc3d","Type":"ContainerStarted","Data":"e70a4a65ad381380a4622f0d9cc993f653871882a54b7f9862fe780a2e4bdc89"} Nov 02 22:52:34.785391 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:34.785363 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:34.785754 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:34.785460 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:34.944435 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:34.944404 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" event={"ID":"c8ae4d29-9676-4fff-a0cb-a26179a61e50","Type":"ContainerStarted","Data":"957dc26c30d4bc5e6415a4a444a1035a639ab434289ae30399999f76db1a6dd6"} Nov 02 22:52:35.054642 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:35.054559 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cqbm6" podStartSLOduration=7.921887891 podStartE2EDuration="21.054545772s" totalImagesPullingTime="13.132657881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:35.053332406 +0000 UTC m=+21.840761367" watchObservedRunningTime="2026-11-02 22:52:35.054545772 +0000 UTC m=+21.841974734" Nov 02 22:52:35.785452 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:35.785283 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:35.785925 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:35.785335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:35.785925 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:35.785524 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:35.785925 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:35.785648 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:35.949791 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:35.949689 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" event={"ID":"584a133a-3ca7-4a1b-88fc-83d98e3abc3d","Type":"ContainerStarted","Data":"d665024c2f1f10b1f5a6abc5bc49ab18db4baac28e654244e1278e99bbec5765"} Nov 02 22:52:36.202796 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:36.202707 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" podStartSLOduration=7.125271643 podStartE2EDuration="22.202695039s" totalImagesPullingTime="15.077423396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:36.155813779 +0000 UTC m=+22.943242740" watchObservedRunningTime="2026-11-02 22:52:36.202695039 +0000 UTC m=+22.990124001" Nov 02 22:52:36.784913 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:36.784886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:36.785071 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:36.784976 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:36.952283 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:36.952255 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:36.952792 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:36.952340 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:36.952792 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:36.952387 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:36.967987 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:36.967963 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:36.974233 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:36.974023 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:52:37.785098 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:37.785065 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:37.785245 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:37.785201 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:37.785302 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:37.785251 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:37.785362 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:37.785343 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:38.785301 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:38.785273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:38.785728 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:38.785369 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:39.784750 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:39.784717 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:39.784929 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:39.784835 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:39.784929 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:39.784913 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:39.785051 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:39.785029 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:40.540034 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:40.539998 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m2smd_52e9c219-9e54-43c4-9dae-7d60c8e33dfe/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:40.784591 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:40.784564 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:40.784748 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:40.784655 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:40.918273 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:40.918199 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m2smd_52e9c219-9e54-43c4-9dae-7d60c8e33dfe/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:41.144853 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:41.144821 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m2smd_52e9c219-9e54-43c4-9dae-7d60c8e33dfe/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:41.787614 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:41.787582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:41.788008 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:41.787687 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:41.788008 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:41.787737 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:41.788008 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:41.787818 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:42.785020 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:42.784993 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:42.785201 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:42.785090 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:43.785402 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:43.785370 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:43.785888 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:43.785438 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:43.785888 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:43.785514 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:43.785888 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:43.785603 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:44.785283 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:44.785253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:44.785446 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:44.785349 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:45.787047 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:45.787023 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:45.787448 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:45.787118 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:45.787448 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:45.787174 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:45.787448 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:45.787252 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:46.785047 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:46.785017 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:46.785349 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:46.785130 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:47.496850 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:47.496820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:47.497344 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:47.496861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:47.497344 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:47.496948 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:47.497344 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:47.496993 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret podName:ad49b548-e79b-43aa-9479-57c1e6d06fdd nodeName:}" failed. No retries permitted until 2026-11-02 22:53:19.496980655 +0000 UTC m=+66.284409608 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret") pod "global-pull-secret-syncer-9b6c4" (UID: "ad49b548-e79b-43aa-9479-57c1e6d06fdd") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:52:47.497344 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:47.496949 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:47.497344 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:47.497071 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs podName:0798910d-3562-4dfe-b069-587510b73467 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:19.49705712 +0000 UTC m=+66.284486074 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs") pod "network-metrics-daemon-zkbf8" (UID: "0798910d-3562-4dfe-b069-587510b73467") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:52:47.597630 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:47.597602 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:47.597768 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:47.597718 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:52:47.597768 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:47.597731 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:52:47.597768 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:47.597739 2574 projected.go:196] Error preparing data for projected volume kube-api-access-7pcfv for pod openshift-network-diagnostics/network-check-target-2wfzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:47.597869 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:47.597777 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv podName:b37eb397-e28e-49c2-a711-a4efd3466679 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:19.597766277 +0000 UTC m=+66.385195218 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcfv" (UniqueName: "kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv") pod "network-check-target-2wfzr" (UID: "b37eb397-e28e-49c2-a711-a4efd3466679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:52:47.787480 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:47.787411 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:47.787619 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:47.787527 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:47.787619 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:47.787577 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:47.787728 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:47.787649 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:48.785036 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:48.785006 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:48.785453 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:48.785092 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:49.786310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:49.786273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:49.786949 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:49.786919 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:49.787446 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:49.787427 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:49.787654 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:49.787637 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:50.784522 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:50.784494 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:50.784676 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:50.784590 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:51.305231 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:51.305202 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2wfzr"] Nov 02 22:52:51.305700 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:51.305326 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:51.305700 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:51.305433 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:51.308283 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:51.308258 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9b6c4"] Nov 02 22:52:51.308400 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:51.308388 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:51.308500 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:51.308478 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:51.309124 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:51.309097 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zkbf8"] Nov 02 22:52:51.309244 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:51.309216 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:51.309346 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:51.309324 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:52.784663 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:52.784629 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:52.785040 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:52.784634 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:52.785040 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:52.784731 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:52.785040 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:52.784808 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:52.785040 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:52.784629 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:52.785040 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:52.784877 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:54.784749 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:54.784571 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:54.785114 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:54.784604 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:54.785114 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:54.784622 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:54.785114 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:54.784974 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkbf8" podUID="0798910d-3562-4dfe-b069-587510b73467" Nov 02 22:52:54.785114 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:54.784901 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b6c4" podUID="ad49b548-e79b-43aa-9479-57c1e6d06fdd" Nov 02 22:52:54.785114 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:52:54.784817 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wfzr" podUID="b37eb397-e28e-49c2-a711-a4efd3466679" Nov 02 22:52:55.958337 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:55.958313 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeReady" Nov 02 22:52:55.958642 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:55.958434 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 22:52:56.064462 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.064428 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wp45h"] Nov 02 22:52:56.089191 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.089144 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.110583 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.110531 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j2jn2"] Nov 02 22:52:56.125472 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.125452 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mn22n"] Nov 02 22:52:56.125610 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.125570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.141648 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.141625 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:52:56.141747 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.141678 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6gjv\"" Nov 02 22:52:56.141747 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.141678 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:52:56.147458 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.147438 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:52:56.147458 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.147448 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wp45h"] Nov 02 22:52:56.147567 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.147488 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j2jn2"] Nov 02 22:52:56.147567 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.147521 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mn22n"] Nov 02 22:52:56.157753 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.157733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7mrv\" (UniqueName: \"kubernetes.io/projected/b8238d30-1c60-4c49-9171-881a3f46f399-kube-api-access-q7mrv\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.157834 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.157758 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75cd1116-7b34-46e7-9502-fa71e12bbd9c-tmp-dir\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.157834 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.157779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75cd1116-7b34-46e7-9502-fa71e12bbd9c-metrics-tls\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.157834 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.157807 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75cd1116-7b34-46e7-9502-fa71e12bbd9c-config-volume\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.157995 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.157867 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8238d30-1c60-4c49-9171-881a3f46f399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.157995 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.157946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtxmf\" (UniqueName: \"kubernetes.io/projected/75cd1116-7b34-46e7-9502-fa71e12bbd9c-kube-api-access-dtxmf\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.157995 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.157977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a4f8b44-11be-4522-806a-87f1e54ba005-cert\") pod \"ingress-canary-mn22n\" (UID: \"2a4f8b44-11be-4522-806a-87f1e54ba005\") " pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:52:56.158090 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.158003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6w4k\" (UniqueName: \"kubernetes.io/projected/2a4f8b44-11be-4522-806a-87f1e54ba005-kube-api-access-r6w4k\") pod \"ingress-canary-mn22n\" (UID: \"2a4f8b44-11be-4522-806a-87f1e54ba005\") " pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:52:56.158124 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.158086 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8238d30-1c60-4c49-9171-881a3f46f399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.158176 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.158127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8238d30-1c60-4c49-9171-881a3f46f399-crio-socket\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.158176 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.158143 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8238d30-1c60-4c49-9171-881a3f46f399-data-volume\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.173907 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.173879 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:52:56.174061 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.174043 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:52:56.174134 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.174098 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vbflc\"" Nov 02 22:52:56.174244 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.174168 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:52:56.174244 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.174176 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:52:56.204575 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.204552 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:52:56.204676 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.204600 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:52:56.204742 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.204694 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:52:56.204742 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.204704 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8zgv4\"" Nov 02 22:52:56.257038 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.257016 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f695d65dd-478hr"] Nov 02 22:52:56.259327 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.259306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75cd1116-7b34-46e7-9502-fa71e12bbd9c-metrics-tls\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.259425 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.259350 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75cd1116-7b34-46e7-9502-fa71e12bbd9c-config-volume\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.259425 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.259368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8238d30-1c60-4c49-9171-881a3f46f399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.259425 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.259386 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtxmf\" (UniqueName: \"kubernetes.io/projected/75cd1116-7b34-46e7-9502-fa71e12bbd9c-kube-api-access-dtxmf\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.259590 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.259573 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a4f8b44-11be-4522-806a-87f1e54ba005-cert\") pod \"ingress-canary-mn22n\" (UID: \"2a4f8b44-11be-4522-806a-87f1e54ba005\") " pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:52:56.259663 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.259610 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6w4k\" (UniqueName: \"kubernetes.io/projected/2a4f8b44-11be-4522-806a-87f1e54ba005-kube-api-access-r6w4k\") pod \"ingress-canary-mn22n\" (UID: \"2a4f8b44-11be-4522-806a-87f1e54ba005\") " pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:52:56.259663 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.259643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8238d30-1c60-4c49-9171-881a3f46f399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.259763 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.259698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8238d30-1c60-4c49-9171-881a3f46f399-crio-socket\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.259763 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.259739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8238d30-1c60-4c49-9171-881a3f46f399-data-volume\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.259988 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.259965 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8238d30-1c60-4c49-9171-881a3f46f399-crio-socket\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.260049 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.260002 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8238d30-1c60-4c49-9171-881a3f46f399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.260049 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.260008 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8238d30-1c60-4c49-9171-881a3f46f399-data-volume\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.260049 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.260014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mrv\" (UniqueName: \"kubernetes.io/projected/b8238d30-1c60-4c49-9171-881a3f46f399-kube-api-access-q7mrv\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.260219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.260064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75cd1116-7b34-46e7-9502-fa71e12bbd9c-tmp-dir\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.260219 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.260118 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75cd1116-7b34-46e7-9502-fa71e12bbd9c-config-volume\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.260357 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.260340 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75cd1116-7b34-46e7-9502-fa71e12bbd9c-tmp-dir\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.263474 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.263446 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75cd1116-7b34-46e7-9502-fa71e12bbd9c-metrics-tls\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.263612 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.263521 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8238d30-1c60-4c49-9171-881a3f46f399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.263678 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.263636 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a4f8b44-11be-4522-806a-87f1e54ba005-cert\") pod \"ingress-canary-mn22n\" (UID: \"2a4f8b44-11be-4522-806a-87f1e54ba005\") " pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:52:56.286133 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.286113 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.312364 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.312341 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtxmf\" (UniqueName: \"kubernetes.io/projected/75cd1116-7b34-46e7-9502-fa71e12bbd9c-kube-api-access-dtxmf\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.312775 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.312754 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6w4k\" (UniqueName: \"kubernetes.io/projected/2a4f8b44-11be-4522-806a-87f1e54ba005-kube-api-access-r6w4k\") pod \"ingress-canary-mn22n\" (UID: \"2a4f8b44-11be-4522-806a-87f1e54ba005\") " pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:52:56.313462 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.313379 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7mrv\" (UniqueName: \"kubernetes.io/projected/b8238d30-1c60-4c49-9171-881a3f46f399-kube-api-access-q7mrv\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.314921 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.314901 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f695d65dd-478hr"] Nov 02 22:52:56.335826 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.335777 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 22:52:56.335960 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.335940 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 22:52:56.336030 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.335995 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 22:52:56.336030 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.335994 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 22:52:56.336140 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.336054 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 22:52:56.336140 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.336103 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 22:52:56.336230 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.336215 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 22:52:56.336301 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.336289 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9lw2l\"" Nov 02 22:52:56.360633 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.360580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.360633 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.360628 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.360749 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.360652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.360749 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.360726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlrjz\" (UniqueName: \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.360844 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.360749 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.360844 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.360811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.360844 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.360833 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aeea33da-f0ef-4912-8ebc-29b48d336448-tmp\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.426012 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.425979 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wp45h" Nov 02 22:52:56.461825 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.461800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrjz\" (UniqueName: \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.461908 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.461838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.461908 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.461855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.462005 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.461986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aeea33da-f0ef-4912-8ebc-29b48d336448-tmp\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.462087 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.462067 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.462217 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.462123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.462217 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.462165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.462407 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.462387 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aeea33da-f0ef-4912-8ebc-29b48d336448-tmp\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.462681 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.462658 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.462785 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.462764 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.462857 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.462783 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.464304 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.464286 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.464758 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.464737 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.486158 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.486117 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:52:56.500239 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.500215 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:52:56.514614 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.514588 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlrjz\" (UniqueName: \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.610583 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.610558 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:52:56.784996 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.784965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:52:56.785114 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.784982 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:52:56.785114 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.785089 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:52:56.834590 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.834565 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:52:56.834590 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.834589 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:52:56.834757 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.834681 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2rjj4\"" Nov 02 22:52:56.834757 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.834694 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5kxql\"" Nov 02 22:52:56.834838 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.834761 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:52:56.834838 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.834791 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n2bfg\"" Nov 02 22:52:56.834898 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.834838 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:52:56.850959 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.850941 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wp45h"] Nov 02 22:52:56.898811 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.898739 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j2jn2"] Nov 02 22:52:56.901019 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:56.900996 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8238d30_1c60_4c49_9171_881a3f46f399.slice/crio-1a76bd241011454f4ec6980e8eaad00d38048b00351566c3049ebc32a954c143 WatchSource:0}: Error finding container 1a76bd241011454f4ec6980e8eaad00d38048b00351566c3049ebc32a954c143: Status 404 returned error can't find the container with id 1a76bd241011454f4ec6980e8eaad00d38048b00351566c3049ebc32a954c143 Nov 02 22:52:56.904962 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.904945 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mn22n"] Nov 02 22:52:56.909818 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:56.909797 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4f8b44_11be_4522_806a_87f1e54ba005.slice/crio-feb8301fde9e6bbef58794904ee0dfbe1e30443ec34482097009f23b2aaa902a WatchSource:0}: Error finding container feb8301fde9e6bbef58794904ee0dfbe1e30443ec34482097009f23b2aaa902a: Status 404 returned error can't find the container with id feb8301fde9e6bbef58794904ee0dfbe1e30443ec34482097009f23b2aaa902a Nov 02 22:52:56.979764 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.979738 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f695d65dd-478hr"] Nov 02 22:52:56.983188 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:52:56.983159 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeea33da_f0ef_4912_8ebc_29b48d336448.slice/crio-9a4df370ea05876a66ed6da79c4b1f7e85906f89b6ffd07d4ba0b8ed637e490e WatchSource:0}: Error finding container 9a4df370ea05876a66ed6da79c4b1f7e85906f89b6ffd07d4ba0b8ed637e490e: Status 404 returned error can't find the container with id 9a4df370ea05876a66ed6da79c4b1f7e85906f89b6ffd07d4ba0b8ed637e490e Nov 02 22:52:56.992832 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.992803 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2jn2" event={"ID":"b8238d30-1c60-4c49-9171-881a3f46f399","Type":"ContainerStarted","Data":"08024c8ff62fbf0931516f791ccd701d9cfc35513b5e38b7d7de0345e1e8419c"} Nov 02 22:52:56.992943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.992856 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2jn2" event={"ID":"b8238d30-1c60-4c49-9171-881a3f46f399","Type":"ContainerStarted","Data":"1a76bd241011454f4ec6980e8eaad00d38048b00351566c3049ebc32a954c143"} Nov 02 22:52:56.993880 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.993853 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mn22n" event={"ID":"2a4f8b44-11be-4522-806a-87f1e54ba005","Type":"ContainerStarted","Data":"feb8301fde9e6bbef58794904ee0dfbe1e30443ec34482097009f23b2aaa902a"} Nov 02 22:52:56.994908 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.994889 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wp45h" event={"ID":"75cd1116-7b34-46e7-9502-fa71e12bbd9c","Type":"ContainerStarted","Data":"ad78eb0d0ea0aca8fdc25a38d1e5b95999d27e52d347a4d33057a06a23f65abb"} Nov 02 22:52:56.995976 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:56.995957 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f695d65dd-478hr" event={"ID":"aeea33da-f0ef-4912-8ebc-29b48d336448","Type":"ContainerStarted","Data":"9a4df370ea05876a66ed6da79c4b1f7e85906f89b6ffd07d4ba0b8ed637e490e"} Nov 02 22:52:59.005715 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:52:59.005660 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2jn2" event={"ID":"b8238d30-1c60-4c49-9171-881a3f46f399","Type":"ContainerStarted","Data":"0417877c93693c9d015ab4ae4ebd0a62966fd5ee996abe56116f3dc0b9b1acfa"} Nov 02 22:53:00.010879 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.010641 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wp45h" event={"ID":"75cd1116-7b34-46e7-9502-fa71e12bbd9c","Type":"ContainerStarted","Data":"e34cdedfe974a3bb2f347f0d74cd918463c3724e6981b043cfc1ebfc799e7168"} Nov 02 22:53:00.011351 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.010917 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wp45h" event={"ID":"75cd1116-7b34-46e7-9502-fa71e12bbd9c","Type":"ContainerStarted","Data":"9a54200362f3adb1ba164b6d0274d7e1905b8364a0a8e6f5c00cc5f5e723a4e2"} Nov 02 22:53:00.011351 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.010988 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wp45h" Nov 02 22:53:00.118683 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.118614 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-wp45h" podStartSLOduration=1.488255296 podStartE2EDuration="4.118596168s" totalImagesPullingTime="2.630340872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:00.11795752 +0000 UTC m=+46.905386482" watchObservedRunningTime="2026-11-02 22:53:00.118596168 +0000 UTC m=+46.906025132" Nov 02 22:53:00.678395 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.678361 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx"] Nov 02 22:53:00.680576 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.680554 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.697407 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.697379 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.697522 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.697496 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.703532 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.697545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d17af01-21f0-48b5-b6b3-834a2e2c7852-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.703532 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.697568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjjh\" (UniqueName: \"kubernetes.io/projected/7d17af01-21f0-48b5-b6b3-834a2e2c7852-kube-api-access-pqjjh\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.730099 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.730070 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:53:00.730292 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.730072 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:53:00.776390 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.775961 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 22:53:00.776390 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.775973 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wkpfp\"" Nov 02 22:53:00.776390 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.776069 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 22:53:00.776390 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.776134 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 22:53:00.777961 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.777881 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx"] Nov 02 22:53:00.779852 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.779703 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn"] Nov 02 22:53:00.782868 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.782844 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4vrnv"] Nov 02 22:53:00.783000 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.782979 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:00.785316 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.785298 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:00.798298 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.798266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.798387 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.798319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d17af01-21f0-48b5-b6b3-834a2e2c7852-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.798387 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.798348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjjh\" (UniqueName: \"kubernetes.io/projected/7d17af01-21f0-48b5-b6b3-834a2e2c7852-kube-api-access-pqjjh\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.798489 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.798422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.799729 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.799704 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d17af01-21f0-48b5-b6b3-834a2e2c7852-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.802959 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.802938 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.803059 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.802992 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.831362 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.831307 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 22:53:00.831679 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.831652 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 22:53:00.831791 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.831679 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mvxcq\"" Nov 02 22:53:00.831791 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.831728 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 22:53:00.852542 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.852518 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjjh\" (UniqueName: \"kubernetes.io/projected/7d17af01-21f0-48b5-b6b3-834a2e2c7852-kube-api-access-pqjjh\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:00.871080 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.870999 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn"] Nov 02 22:53:00.899749 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.899689 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:00.899749 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.899729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-root\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:00.899908 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.899758 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:00.899908 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.899838 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:00.899908 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.899875 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:00.899908 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.899903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe3683b5-cfe1-408e-be5b-5a81c36304c4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:00.900066 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.900024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-tls\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:00.900066 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.900058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-sys\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:00.900125 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.900080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2j4\" (UniqueName: \"kubernetes.io/projected/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-api-access-4f2j4\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:00.900125 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.900106 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-accelerators-collector-config\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:00.900217 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.900124 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-metrics-client-ca\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:00.900217 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.900138 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-textfile\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:00.900217 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.900197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:00.900217 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.900218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqkgp\" (UniqueName: \"kubernetes.io/projected/41883b9a-e520-4e4d-8753-4512e53ff141-kube-api-access-pqkgp\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:00.900357 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.900251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-wtmp\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:00.931794 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.931718 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 22:53:00.931794 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.931734 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 22:53:00.931974 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.931918 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rnj8n\"" Nov 02 22:53:00.932252 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.932232 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 22:53:00.996641 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:00.996621 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:01.000619 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.000593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-tls\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.000725 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.000653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-sys\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.000725 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.000680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2j4\" (UniqueName: \"kubernetes.io/projected/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-api-access-4f2j4\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.000936 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.000892 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-sys\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.001310 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.000893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-accelerators-collector-config\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.001404 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.001343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-metrics-client-ca\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.001404 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.001383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-textfile\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.002262 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.002238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.002372 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.002299 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqkgp\" (UniqueName: \"kubernetes.io/projected/41883b9a-e520-4e4d-8753-4512e53ff141-kube-api-access-pqkgp\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.002372 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.002344 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-wtmp\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.002481 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.002381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.002481 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.002412 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-root\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.002481 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.002476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.002615 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.002510 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.002615 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.002540 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.002615 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.002598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe3683b5-cfe1-408e-be5b-5a81c36304c4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.002748 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.002697 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-accelerators-collector-config\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.003052 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.003020 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe3683b5-cfe1-408e-be5b-5a81c36304c4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.003616 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.003594 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.005527 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.004397 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.005527 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.005132 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-tls\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.005527 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.005251 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-metrics-client-ca\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.005527 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.005481 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-wtmp\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.005799 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.005771 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-textfile\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.006480 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.006447 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.007258 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.007234 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-root\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.010168 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.010061 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.011445 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.011423 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.015610 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.015579 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mn22n" event={"ID":"2a4f8b44-11be-4522-806a-87f1e54ba005","Type":"ContainerStarted","Data":"68086c1193b306868af0a27455451c1d277b2f4e768e8397ee7dcdfc11a27a2e"} Nov 02 22:53:01.018623 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.018516 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j2jn2" event={"ID":"b8238d30-1c60-4c49-9171-881a3f46f399","Type":"ContainerStarted","Data":"0743ed96cd4e3dd2ff01548701c3419edb0d5364b1d06175d579d198bd1b1eb9"} Nov 02 22:53:01.053585 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.053556 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2j4\" (UniqueName: \"kubernetes.io/projected/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-api-access-4f2j4\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.083654 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.083626 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqkgp\" (UniqueName: \"kubernetes.io/projected/41883b9a-e520-4e4d-8753-4512e53ff141-kube-api-access-pqkgp\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.101223 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.101178 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:01.107236 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.107213 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:01.116260 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:53:01.116238 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41883b9a_e520_4e4d_8753_4512e53ff141.slice/crio-c39e16201ad053f0f82704ec6b09adaa194cc5ae5f191410f2a81b7961cbe360 WatchSource:0}: Error finding container c39e16201ad053f0f82704ec6b09adaa194cc5ae5f191410f2a81b7961cbe360: Status 404 returned error can't find the container with id c39e16201ad053f0f82704ec6b09adaa194cc5ae5f191410f2a81b7961cbe360 Nov 02 22:53:01.148096 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.148046 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j2jn2" podStartSLOduration=1.5413681000000001 podStartE2EDuration="5.147849023s" totalImagesPullingTime="3.606480923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:01.147108019 +0000 UTC m=+47.934536982" watchObservedRunningTime="2026-11-02 22:53:01.147849023 +0000 UTC m=+47.935277986" Nov 02 22:53:01.252795 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.252723 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mn22n" podStartSLOduration=1.223625985 podStartE2EDuration="5.252709643s" totalImagesPullingTime="4.029083658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:01.252530445 +0000 UTC m=+48.039959407" watchObservedRunningTime="2026-11-02 22:53:01.252709643 +0000 UTC m=+48.040138605" Nov 02 22:53:01.413719 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.413680 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx"] Nov 02 22:53:01.417431 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:53:01.417406 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d17af01_21f0_48b5_b6b3_834a2e2c7852.slice/crio-c613782942a74e0881d589b6c646cef4bdf2ca31db51a2230fd393fb48897d60 WatchSource:0}: Error finding container c613782942a74e0881d589b6c646cef4bdf2ca31db51a2230fd393fb48897d60: Status 404 returned error can't find the container with id c613782942a74e0881d589b6c646cef4bdf2ca31db51a2230fd393fb48897d60 Nov 02 22:53:01.478186 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:01.478158 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn"] Nov 02 22:53:01.484124 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:53:01.484103 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3683b5_cfe1_408e_be5b_5a81c36304c4.slice/crio-a23e71108770675e264786c5b980bbea88535472d6caaf6a2a8ecfcbd8f178f3 WatchSource:0}: Error finding container a23e71108770675e264786c5b980bbea88535472d6caaf6a2a8ecfcbd8f178f3: Status 404 returned error can't find the container with id a23e71108770675e264786c5b980bbea88535472d6caaf6a2a8ecfcbd8f178f3 Nov 02 22:53:02.022744 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:02.022708 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f695d65dd-478hr" event={"ID":"aeea33da-f0ef-4912-8ebc-29b48d336448","Type":"ContainerStarted","Data":"64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb"} Nov 02 22:53:02.023868 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:02.023839 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4vrnv" event={"ID":"41883b9a-e520-4e4d-8753-4512e53ff141","Type":"ContainerStarted","Data":"c39e16201ad053f0f82704ec6b09adaa194cc5ae5f191410f2a81b7961cbe360"} Nov 02 22:53:02.024895 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:02.024878 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" event={"ID":"fe3683b5-cfe1-408e-be5b-5a81c36304c4","Type":"ContainerStarted","Data":"a23e71108770675e264786c5b980bbea88535472d6caaf6a2a8ecfcbd8f178f3"} Nov 02 22:53:02.026626 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:02.026595 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" event={"ID":"7d17af01-21f0-48b5-b6b3-834a2e2c7852","Type":"ContainerStarted","Data":"cb04fa48308ac8a6fb7e9d6e6324eca440839c40503059558c5f46fff1f6ee71"} Nov 02 22:53:02.026708 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:02.026648 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" event={"ID":"7d17af01-21f0-48b5-b6b3-834a2e2c7852","Type":"ContainerStarted","Data":"a25409a5067561006af9b9366ec42db09bd6bf67833c8ce57b66e8c92f155ac7"} Nov 02 22:53:02.026708 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:02.026668 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" event={"ID":"7d17af01-21f0-48b5-b6b3-834a2e2c7852","Type":"ContainerStarted","Data":"c613782942a74e0881d589b6c646cef4bdf2ca31db51a2230fd393fb48897d60"} Nov 02 22:53:02.132604 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:02.132561 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f695d65dd-478hr" podStartSLOduration=2.005456924 podStartE2EDuration="6.132548935s" totalImagesPullingTime="4.127092011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:02.132238362 +0000 UTC m=+48.919667324" watchObservedRunningTime="2026-11-02 22:53:02.132548935 +0000 UTC m=+48.919977897" Nov 02 22:53:03.032484 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.032451 2574 generic.go:356] "Generic (PLEG): container finished" podID="41883b9a-e520-4e4d-8753-4512e53ff141" containerID="bf2d19b016df5edaef14cc792e217293d5674965e9e78ee136624e87f8d3cc9f" exitCode=0 Nov 02 22:53:03.032875 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.032534 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4vrnv" event={"ID":"41883b9a-e520-4e4d-8753-4512e53ff141","Type":"ContainerDied","Data":"bf2d19b016df5edaef14cc792e217293d5674965e9e78ee136624e87f8d3cc9f"} Nov 02 22:53:03.347313 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.347288 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b8df694dc-d54td"] Nov 02 22:53:03.350289 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.350263 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.399291 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.399269 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 22:53:03.399411 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.399328 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 22:53:03.399411 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.399358 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 22:53:03.399411 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.399269 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m8ggg\"" Nov 02 22:53:03.399411 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.399268 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 22:53:03.399626 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.399424 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b9iueu8j41tm9\"" Nov 02 22:53:03.399626 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.399479 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 22:53:03.406960 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.406938 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b8df694dc-d54td"] Nov 02 22:53:03.421192 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.421107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d158803-71d8-447b-8e4d-e73287682505-metrics-client-ca\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.421192 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.421142 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.421361 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.421231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d158803-71d8-447b-8e4d-e73287682505-tmp\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.421361 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.421285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.421459 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.421371 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-tls\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.421459 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.421444 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxx5\" (UniqueName: \"kubernetes.io/projected/4d158803-71d8-447b-8e4d-e73287682505-kube-api-access-nqxx5\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.421556 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.421472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.421556 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.421531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.421646 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.421567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-grpc-tls\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.522888 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.522856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.523056 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.522906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-tls\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.523056 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.522945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxx5\" (UniqueName: \"kubernetes.io/projected/4d158803-71d8-447b-8e4d-e73287682505-kube-api-access-nqxx5\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.523056 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.522963 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.523056 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.522984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.523056 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.523004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-grpc-tls\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.523056 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.523035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d158803-71d8-447b-8e4d-e73287682505-metrics-client-ca\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.523372 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.523063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.523372 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.523088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d158803-71d8-447b-8e4d-e73287682505-tmp\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.523582 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.523507 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d158803-71d8-447b-8e4d-e73287682505-tmp\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.524001 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.523971 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d158803-71d8-447b-8e4d-e73287682505-metrics-client-ca\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.525974 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.525948 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-tls\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.526072 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.526004 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.526708 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.526658 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.526788 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.526733 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.526788 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.526779 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.527081 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.527065 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-grpc-tls\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.576648 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.576623 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxx5\" (UniqueName: \"kubernetes.io/projected/4d158803-71d8-447b-8e4d-e73287682505-kube-api-access-nqxx5\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:03.689247 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:03.689225 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:04.037740 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:04.037707 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4vrnv" event={"ID":"41883b9a-e520-4e4d-8753-4512e53ff141","Type":"ContainerStarted","Data":"0cfb8b75d99f9c48dc20a554376485bd9e871778c4ae12904f3c184c43762a81"} Nov 02 22:53:04.038211 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:04.037782 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4vrnv" event={"ID":"41883b9a-e520-4e4d-8753-4512e53ff141","Type":"ContainerStarted","Data":"dc0efcb93c85f4f7ed93801198969f05301213e55bec679286a5fa7591344351"} Nov 02 22:53:04.043789 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:04.040615 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" event={"ID":"fe3683b5-cfe1-408e-be5b-5a81c36304c4","Type":"ContainerStarted","Data":"fa0ce65a11fb048d6eee8fdfd567b184aada98c26205fe6433efb7e619800bbd"} Nov 02 22:53:04.043789 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:04.040719 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" event={"ID":"fe3683b5-cfe1-408e-be5b-5a81c36304c4","Type":"ContainerStarted","Data":"71cea1d4f218aad522075b93532625ce2d8994e67d9a5b1e3e8e73c3b66fd58d"} Nov 02 22:53:04.043789 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:04.040755 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" event={"ID":"fe3683b5-cfe1-408e-be5b-5a81c36304c4","Type":"ContainerStarted","Data":"21fdc39527fa309abf621f0ef39caaa59db40432a8fa40327520dd849efa3e46"} Nov 02 22:53:04.044053 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:04.044028 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" event={"ID":"7d17af01-21f0-48b5-b6b3-834a2e2c7852","Type":"ContainerStarted","Data":"912ec8544db5b5dc6729bdbd7e8e9f8b887a953d5916c441c9659ad54f6fce61"} Nov 02 22:53:04.145989 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:04.145969 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b8df694dc-d54td"] Nov 02 22:53:04.147805 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:04.147762 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4vrnv" podStartSLOduration=2.342312789 podStartE2EDuration="4.147751207s" totalImagesPullingTime="1.7253783s" totalInitContainerRuntime="80.060118ms" isStatefulPod=true podCreationTimestamp="2026-11-02 22:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:04.146685053 +0000 UTC m=+50.934114015" watchObservedRunningTime="2026-11-02 22:53:04.147751207 +0000 UTC m=+50.935180185" Nov 02 22:53:04.254508 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:04.254452 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" podStartSLOduration=2.429023485 podStartE2EDuration="4.254436465s" totalImagesPullingTime="1.82541298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:04.253716233 +0000 UTC m=+51.041145196" watchObservedRunningTime="2026-11-02 22:53:04.254436465 +0000 UTC m=+51.041865426" Nov 02 22:53:04.362650 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:04.362541 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" podStartSLOduration=2.320756797 podStartE2EDuration="4.362526687s" totalImagesPullingTime="2.04176989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:04.361799679 +0000 UTC m=+51.149228642" watchObservedRunningTime="2026-11-02 22:53:04.362526687 +0000 UTC m=+51.149955649" Nov 02 22:53:05.048561 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.048523 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" event={"ID":"4d158803-71d8-447b-8e4d-e73287682505","Type":"ContainerStarted","Data":"1214330d59b32972f40ac043f7fe5926ff65fbdee121a7cd3a0f93b3bec8232e"} Nov 02 22:53:05.433804 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.433730 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz"] Nov 02 22:53:05.436568 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.436553 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:05.487452 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.487427 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 22:53:05.487964 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.487945 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r7xnr\"" Nov 02 22:53:05.488510 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.488488 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz"] Nov 02 22:53:05.541651 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.541622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e26c39e8-2ffc-4179-8498-76167f635e9a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zhjrz\" (UID: \"e26c39e8-2ffc-4179-8498-76167f635e9a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:05.541735 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.541661 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pnv7\" (UniqueName: \"kubernetes.io/projected/e26c39e8-2ffc-4179-8498-76167f635e9a-kube-api-access-4pnv7\") pod \"monitoring-plugin-77f4f76594-zhjrz\" (UID: \"e26c39e8-2ffc-4179-8498-76167f635e9a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:05.642110 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.642081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e26c39e8-2ffc-4179-8498-76167f635e9a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zhjrz\" (UID: \"e26c39e8-2ffc-4179-8498-76167f635e9a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:05.642237 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.642124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pnv7\" (UniqueName: \"kubernetes.io/projected/e26c39e8-2ffc-4179-8498-76167f635e9a-kube-api-access-4pnv7\") pod \"monitoring-plugin-77f4f76594-zhjrz\" (UID: \"e26c39e8-2ffc-4179-8498-76167f635e9a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:05.644479 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.644457 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e26c39e8-2ffc-4179-8498-76167f635e9a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zhjrz\" (UID: \"e26c39e8-2ffc-4179-8498-76167f635e9a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:05.695580 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.695523 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pnv7\" (UniqueName: \"kubernetes.io/projected/e26c39e8-2ffc-4179-8498-76167f635e9a-kube-api-access-4pnv7\") pod \"monitoring-plugin-77f4f76594-zhjrz\" (UID: \"e26c39e8-2ffc-4179-8498-76167f635e9a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:05.747058 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:05.747036 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:06.106404 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:06.106383 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz"] Nov 02 22:53:06.109013 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:53:06.108972 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26c39e8_2ffc_4179_8498_76167f635e9a.slice/crio-a79d7d3a32c515e2fc0fe618cc824ac64767af250e79f9f8f7f4ae8d94acda0b WatchSource:0}: Error finding container a79d7d3a32c515e2fc0fe618cc824ac64767af250e79f9f8f7f4ae8d94acda0b: Status 404 returned error can't find the container with id a79d7d3a32c515e2fc0fe618cc824ac64767af250e79f9f8f7f4ae8d94acda0b Nov 02 22:53:06.611576 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:06.611536 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:06.611742 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:06.611595 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:06.616222 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:06.616198 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:07.057705 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:07.057666 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" event={"ID":"e26c39e8-2ffc-4179-8498-76167f635e9a","Type":"ContainerStarted","Data":"a79d7d3a32c515e2fc0fe618cc824ac64767af250e79f9f8f7f4ae8d94acda0b"} Nov 02 22:53:07.062775 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:07.062743 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:08.064300 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:08.063913 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" event={"ID":"4d158803-71d8-447b-8e4d-e73287682505","Type":"ContainerStarted","Data":"5f4569eea0492ecad5faa8cbbe24d7a07824aca956b71bf733b1722737d29142"} Nov 02 22:53:08.064300 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:08.064002 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" event={"ID":"4d158803-71d8-447b-8e4d-e73287682505","Type":"ContainerStarted","Data":"ba3088782daf1db9198628b137f203a40791bb365168bf63251fee1eb7b9f983"} Nov 02 22:53:08.966385 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:08.966356 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:09.068490 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.068461 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" event={"ID":"4d158803-71d8-447b-8e4d-e73287682505","Type":"ContainerStarted","Data":"2c024eb3f436d69d781ae0d70fda679b7df52da3f73e2cee4868ccad41313b31"} Nov 02 22:53:09.069781 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.069760 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" event={"ID":"e26c39e8-2ffc-4179-8498-76167f635e9a","Type":"ContainerStarted","Data":"d76bbf08ec103902c010d646eaabd8596e676b3b29a27ebcf0cf68ce411ae918"} Nov 02 22:53:09.069942 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.069926 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:09.074411 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.074381 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:09.279850 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.279790 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" podStartSLOduration=2.219389569 podStartE2EDuration="4.279770043s" totalImagesPullingTime="2.060380474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:09.279125914 +0000 UTC m=+56.066554875" watchObservedRunningTime="2026-11-02 22:53:09.279770043 +0000 UTC m=+56.067199210" Nov 02 22:53:09.545382 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.545361 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:53:09.548999 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.548981 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.598555 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.598508 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 22:53:09.598812 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.598791 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 22:53:09.598812 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.598792 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 22:53:09.598952 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.598793 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 22:53:09.598952 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.598891 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 22:53:09.628512 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.628492 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 22:53:09.628512 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.628495 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 22:53:09.644686 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.644667 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 22:53:09.644754 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.644707 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 22:53:09.645008 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.644994 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-75wn4\"" Nov 02 22:53:09.680191 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680166 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-config\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680304 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680304 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680304 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680431 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680431 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680431 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680431 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680388 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680569 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680466 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680569 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680569 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680569 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680682 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680682 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680682 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680682 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbf65\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-kube-api-access-qbf65\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680794 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680732 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680794 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680775 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680848 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.680882 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.680852 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.692048 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.692028 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 22:53:09.692172 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.692143 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 22:53:09.694141 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.694124 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:53:09.738685 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.738655 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 22:53:09.739005 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.738973 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 22:53:09.739111 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.739011 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 22:53:09.739111 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.739048 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 22:53:09.739214 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.739120 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-tedh3m60u5r0\"" Nov 02 22:53:09.781614 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.781614 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.781614 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781604 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.781798 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.781798 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.781798 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbf65\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-kube-api-access-qbf65\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.781798 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.781798 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781796 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782022 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782022 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782022 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781894 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-config\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782022 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782022 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781959 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782022 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.781988 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782022 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.782012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782417 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.782037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782417 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.782065 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782417 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.782088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782417 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.782132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782417 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.782176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782675 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.782529 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.782675 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.782626 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.783293 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.783267 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.783915 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.783890 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.784543 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.784508 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-config\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.785122 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.784807 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.785122 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.784833 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.785122 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.784812 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.785122 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.785102 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.785396 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.785173 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.785796 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.785770 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.786374 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.786347 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.786622 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.786598 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.786862 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.786835 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.786943 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.786886 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.787392 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.787368 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.788221 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.788189 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.788312 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.788255 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.788531 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.788502 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.834697 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.834674 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbf65\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-kube-api-access-qbf65\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:09.895793 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:09.895769 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:10.020764 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:10.020734 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wp45h" Nov 02 22:53:10.076235 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:10.076171 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" event={"ID":"4d158803-71d8-447b-8e4d-e73287682505","Type":"ContainerStarted","Data":"9a35c9690b6bd67cc1ed9f4bd9a5b1fd920f465b9e1b04e16cd7f5a7ce4e4023"} Nov 02 22:53:10.076235 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:10.076227 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" event={"ID":"4d158803-71d8-447b-8e4d-e73287682505","Type":"ContainerStarted","Data":"65aa9d7d4379fdd929fc9383b711de0ef5f141c41c7bda7c8ee7bcdbd71c4e0c"} Nov 02 22:53:10.076638 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:10.076253 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" event={"ID":"4d158803-71d8-447b-8e4d-e73287682505","Type":"ContainerStarted","Data":"34f3a5b0d25486f0327f4d412b5099d2d136260433c62fba00b21da652e99f11"} Nov 02 22:53:10.238444 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:10.238391 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" podStartSLOduration=2.036423977 podStartE2EDuration="7.238376672s" totalImagesPullingTime="5.201952695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:10.236965982 +0000 UTC m=+57.024394964" watchObservedRunningTime="2026-11-02 22:53:10.238376672 +0000 UTC m=+57.025805638" Nov 02 22:53:10.315753 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:10.315721 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:53:10.340191 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:53:10.340141 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a9d6b0_dd41_4d54_bf35_110826270d66.slice/crio-07fde3b76be6a14be4598fa33edaa6ecf159820c47afd3b0aea63a609037b676 WatchSource:0}: Error finding container 07fde3b76be6a14be4598fa33edaa6ecf159820c47afd3b0aea63a609037b676: Status 404 returned error can't find the container with id 07fde3b76be6a14be4598fa33edaa6ecf159820c47afd3b0aea63a609037b676 Nov 02 22:53:11.080443 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:11.080398 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerStarted","Data":"07fde3b76be6a14be4598fa33edaa6ecf159820c47afd3b0aea63a609037b676"} Nov 02 22:53:11.080859 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:11.080841 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:12.084681 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:12.084620 2574 generic.go:356] "Generic (PLEG): container finished" podID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerID="89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c" exitCode=0 Nov 02 22:53:12.085017 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:12.084705 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerDied","Data":"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c"} Nov 02 22:53:16.106474 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:16.106433 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerStarted","Data":"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9"} Nov 02 22:53:16.106865 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:16.106505 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerStarted","Data":"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de"} Nov 02 22:53:16.106865 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:16.106533 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerStarted","Data":"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af"} Nov 02 22:53:16.106865 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:16.106560 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerStarted","Data":"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407"} Nov 02 22:53:16.106865 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:16.106585 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerStarted","Data":"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a"} Nov 02 22:53:16.106865 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:16.106606 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerStarted","Data":"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134"} Nov 02 22:53:16.251646 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:16.251556 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.094291491 podStartE2EDuration="7.251537268s" totalImagesPullingTime="5.141120171s" totalInitContainerRuntime="16.125606ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:16.223910444 +0000 UTC m=+63.011339417" watchObservedRunningTime="2026-11-02 22:53:16.251537268 +0000 UTC m=+63.038966231" Nov 02 22:53:17.091927 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:17.091902 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:18.876396 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:18.876358 2574 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f695d65dd-478hr"] Nov 02 22:53:19.572941 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.572906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:53:19.573099 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.572948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:53:19.621104 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.621075 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:53:19.621218 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.621128 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:53:19.626329 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.626310 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:53:19.626409 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.626310 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:53:19.674252 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.674230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:53:19.714554 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.714532 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5kxql\"" Nov 02 22:53:19.721830 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.721807 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:53:19.722841 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.722826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:53:19.772798 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.772767 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:53:19.779868 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.779808 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:53:19.896835 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.896764 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:19.956446 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.956419 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n2bfg\"" Nov 02 22:53:19.959518 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.959501 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:53:19.973520 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.973501 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2rjj4\"" Nov 02 22:53:19.976935 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:19.976917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:53:20.085341 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:20.085297 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zkbf8"] Nov 02 22:53:20.087398 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:53:20.087363 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0798910d_3562_4dfe_b069_587510b73467.slice/crio-1c4b324d7b7cf8c78a0eeb32bfedc7b0aa61951ecd900ffba76c224cad5d0da7 WatchSource:0}: Error finding container 1c4b324d7b7cf8c78a0eeb32bfedc7b0aa61951ecd900ffba76c224cad5d0da7: Status 404 returned error can't find the container with id 1c4b324d7b7cf8c78a0eeb32bfedc7b0aa61951ecd900ffba76c224cad5d0da7 Nov 02 22:53:20.120835 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:20.120803 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkbf8" event={"ID":"0798910d-3562-4dfe-b069-587510b73467","Type":"ContainerStarted","Data":"1c4b324d7b7cf8c78a0eeb32bfedc7b0aa61951ecd900ffba76c224cad5d0da7"} Nov 02 22:53:20.325373 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:20.325348 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9b6c4"] Nov 02 22:53:20.327222 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:53:20.327193 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad49b548_e79b_43aa_9479_57c1e6d06fdd.slice/crio-84552e920a03143762c5602455e2ccde8a705651a80b46572125592ac6840fec WatchSource:0}: Error finding container 84552e920a03143762c5602455e2ccde8a705651a80b46572125592ac6840fec: Status 404 returned error can't find the container with id 84552e920a03143762c5602455e2ccde8a705651a80b46572125592ac6840fec Nov 02 22:53:20.347501 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:20.347412 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2wfzr"] Nov 02 22:53:20.349632 ip-10-0-137-4 kubenswrapper[2574]: W1102 22:53:20.349609 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37eb397_e28e_49c2_a711_a4efd3466679.slice/crio-270a07fcaa569ce34c85b7e36e5da97c8ef7ce2d616184266fc32999c494a74d WatchSource:0}: Error finding container 270a07fcaa569ce34c85b7e36e5da97c8ef7ce2d616184266fc32999c494a74d: Status 404 returned error can't find the container with id 270a07fcaa569ce34c85b7e36e5da97c8ef7ce2d616184266fc32999c494a74d Nov 02 22:53:21.125824 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:21.125783 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9b6c4" event={"ID":"ad49b548-e79b-43aa-9479-57c1e6d06fdd","Type":"ContainerStarted","Data":"84552e920a03143762c5602455e2ccde8a705651a80b46572125592ac6840fec"} Nov 02 22:53:21.126922 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:21.126898 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2wfzr" event={"ID":"b37eb397-e28e-49c2-a711-a4efd3466679","Type":"ContainerStarted","Data":"270a07fcaa569ce34c85b7e36e5da97c8ef7ce2d616184266fc32999c494a74d"} Nov 02 22:53:22.134215 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:22.134114 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkbf8" event={"ID":"0798910d-3562-4dfe-b069-587510b73467","Type":"ContainerStarted","Data":"b74ee2634852cb63cc07715924fd735e9015c9446c7a5eb956414ab7cd4066e3"} Nov 02 22:53:22.134215 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:22.134202 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkbf8" event={"ID":"0798910d-3562-4dfe-b069-587510b73467","Type":"ContainerStarted","Data":"6271bce48b94db17b0b959c483479b468fed54bc83e0d347f28bd2631fa1a055"} Nov 02 22:53:22.240654 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:22.240594 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zkbf8" podStartSLOduration=66.48292338 podStartE2EDuration="1m8.240573589s" totalImagesPullingTime="1.757650209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:22.23987409 +0000 UTC m=+69.027303052" watchObservedRunningTime="2026-11-02 22:53:22.240573589 +0000 UTC m=+69.028002563" Nov 02 22:53:25.148086 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:25.148044 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2wfzr" event={"ID":"b37eb397-e28e-49c2-a711-a4efd3466679","Type":"ContainerStarted","Data":"4bec4ed015a8ae19eebddd05a53714bdd960aed6e032982519cb086160b7ae66"} Nov 02 22:53:25.148571 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:25.148167 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:53:25.256107 ip-10-0-137-4 kubenswrapper[2574]: I1102 22:53:25.256041 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2wfzr" podStartSLOduration=67.390195733 podStartE2EDuration="1m11.256024384s" totalImagesPullingTime="3.865828651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:25.25552265 +0000 UTC m=+72.042951611" watchObservedRunningTime="2026-11-02 22:53:25.256024384 +0000 UTC m=+72.043453348" Nov 02 22:53:26.513562 ip-10-0-137-4 kubenswrapper[2574]: E1102 22:53:26.513532 2574 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:53:26.529019 ip-10-0-137-4 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 22:53:26.551490 ip-10-0-137-4 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 22:53:26.551658 ip-10-0-137-4 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 22:53:26.551826 ip-10-0-137-4 systemd[1]: kubelet.service: Consumed 4.177s CPU time. Nov 02 22:53:26.562845 ip-10-0-137-4 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:53:26.622933 ip-10-0-137-4 kubenswrapper[5672]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:53:26.622933 ip-10-0-137-4 kubenswrapper[5672]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:53:26.622933 ip-10-0-137-4 kubenswrapper[5672]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:53:26.622933 ip-10-0-137-4 kubenswrapper[5672]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:53:26.624933 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624858 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:53:26.624933 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624933 5672 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624938 5672 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624941 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624943 5672 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624947 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624950 5672 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624953 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624956 5672 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624959 5672 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624962 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624964 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624967 5672 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624970 5672 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624972 5672 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624975 5672 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624977 5672 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624980 5672 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624983 5672 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624986 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624989 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:53:26.625000 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624991 5672 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624994 5672 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.624997 5672 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625000 5672 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625003 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625006 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625009 5672 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625012 5672 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625015 5672 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625018 5672 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625020 5672 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625023 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625026 5672 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625028 5672 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625031 5672 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625034 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625036 5672 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625039 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625042 5672 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625045 5672 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:53:26.625537 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625048 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625051 5672 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625053 5672 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625056 5672 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625058 5672 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625061 5672 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625063 5672 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625066 5672 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625069 5672 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625071 5672 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625075 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625078 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625080 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625085 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625088 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625090 5672 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625093 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625095 5672 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625098 5672 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625100 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:53:26.626021 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625103 5672 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625105 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625108 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625110 5672 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625113 5672 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625116 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625118 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625121 5672 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625123 5672 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625126 5672 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625129 5672 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625132 5672 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625135 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625137 5672 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625140 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625143 5672 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625164 5672 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625167 5672 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625171 5672 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:53:26.626533 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625175 5672 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625178 5672 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625182 5672 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625185 5672 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625188 5672 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625190 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625194 5672 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625197 5672 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625201 5672 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625205 5672 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625208 5672 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625211 5672 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625214 5672 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625217 5672 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625220 5672 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625223 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625226 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625229 5672 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625231 5672 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:53:26.627023 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625234 5672 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625236 5672 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625239 5672 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625241 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625244 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625248 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625252 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625978 5672 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625984 5672 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625987 5672 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625990 5672 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625993 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625996 5672 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.625998 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626001 5672 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626004 5672 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626007 5672 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626010 5672 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626012 5672 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:53:26.627509 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626016 5672 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626018 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626021 5672 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626025 5672 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626028 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626030 5672 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626033 5672 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626036 5672 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626039 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626042 5672 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626044 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626047 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626049 5672 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626052 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626055 5672 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626058 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626060 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626063 5672 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626065 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626069 5672 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:53:26.627997 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626072 5672 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626075 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626078 5672 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626080 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626083 5672 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626086 5672 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626088 5672 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626091 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626093 5672 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626096 5672 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626099 5672 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626101 5672 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626103 5672 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626106 5672 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626109 5672 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626112 5672 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626114 5672 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626117 5672 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626120 5672 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626123 5672 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:53:26.628498 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626125 5672 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626128 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626132 5672 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626134 5672 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626137 5672 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626139 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626142 5672 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626158 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626162 5672 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626164 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626167 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626170 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626174 5672 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626178 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626182 5672 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626185 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626188 5672 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626191 5672 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:53:26.628993 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626193 5672 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626196 5672 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626199 5672 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626201 5672 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626204 5672 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626206 5672 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626209 5672 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626212 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626215 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626218 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626220 5672 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626223 5672 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626226 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626228 5672 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626232 5672 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626236 5672 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626239 5672 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626242 5672 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626245 5672 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:53:26.629444 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626247 5672 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626250 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626252 5672 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626255 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626258 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626260 5672 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626265 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626268 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626270 5672 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626273 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626277 5672 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626280 5672 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626283 5672 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626285 5672 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626298 5672 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626301 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626304 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626752 5672 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626757 5672 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626759 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:53:26.629910 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626762 5672 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626765 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626768 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626771 5672 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626774 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626777 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626779 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626782 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626785 5672 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626787 5672 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626790 5672 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626793 5672 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626796 5672 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626798 5672 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626801 5672 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626804 5672 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626806 5672 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626809 5672 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626813 5672 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:53:26.630404 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626816 5672 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626819 5672 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626829 5672 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626833 5672 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626836 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626838 5672 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626841 5672 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626844 5672 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626846 5672 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626849 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626852 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626854 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626857 5672 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626860 5672 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626863 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626865 5672 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626868 5672 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626870 5672 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626873 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626875 5672 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:53:26.630898 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626878 5672 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626880 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626883 5672 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626885 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626888 5672 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626890 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626893 5672 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626895 5672 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626898 5672 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626901 5672 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626903 5672 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626911 5672 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626917 5672 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626920 5672 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626923 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626926 5672 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626928 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626931 5672 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626934 5672 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626936 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:53:26.631668 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626939 5672 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626943 5672 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626946 5672 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626948 5672 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626951 5672 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626953 5672 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626956 5672 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626959 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626961 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626964 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626966 5672 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626969 5672 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626972 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626976 5672 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626980 5672 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626984 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626987 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626993 5672 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.626996 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:53:26.632350 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627003 5672 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627008 5672 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627012 5672 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627014 5672 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627019 5672 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627024 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627027 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627030 5672 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627032 5672 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627036 5672 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627039 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627041 5672 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627044 5672 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627047 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627049 5672 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627052 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627055 5672 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627057 5672 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627060 5672 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:53:26.632806 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627063 5672 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627065 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627068 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627070 5672 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627073 5672 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627075 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627189 5672 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627203 5672 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627210 5672 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627215 5672 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627219 5672 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627223 5672 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627227 5672 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627231 5672 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627234 5672 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627238 5672 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627241 5672 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627246 5672 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627251 5672 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627254 5672 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627257 5672 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627260 5672 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627263 5672 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:53:26.633295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627267 5672 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627271 5672 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627274 5672 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627277 5672 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627280 5672 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627283 5672 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627288 5672 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627291 5672 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627294 5672 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627297 5672 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627300 5672 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627303 5672 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627306 5672 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627310 5672 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627313 5672 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627317 5672 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627320 5672 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627323 5672 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627326 5672 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627329 5672 flags.go:64] FLAG: --enable-server="true" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627332 5672 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627337 5672 flags.go:64] FLAG: --event-burst="100" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627340 5672 flags.go:64] FLAG: --event-qps="50" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627343 5672 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:53:26.633873 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627346 5672 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627349 5672 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627354 5672 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627359 5672 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627362 5672 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627365 5672 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627368 5672 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627371 5672 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627374 5672 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627378 5672 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627381 5672 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627384 5672 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627387 5672 flags.go:64] FLAG: --feature-gates="" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627391 5672 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627395 5672 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627398 5672 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627401 5672 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627404 5672 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627407 5672 flags.go:64] FLAG: --help="false" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627410 5672 flags.go:64] FLAG: --hostname-override="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627414 5672 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627417 5672 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627420 5672 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:53:26.634479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627423 5672 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627426 5672 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627429 5672 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627432 5672 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627435 5672 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627438 5672 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627441 5672 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627444 5672 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627447 5672 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627450 5672 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627453 5672 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627458 5672 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627462 5672 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627465 5672 flags.go:64] FLAG: --lock-file="" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627468 5672 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627471 5672 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627474 5672 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627480 5672 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627483 5672 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627487 5672 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627490 5672 flags.go:64] FLAG: --logging-format="text" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627493 5672 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627497 5672 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627500 5672 flags.go:64] FLAG: --manifest-url="" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627503 5672 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:53:26.635030 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627508 5672 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627511 5672 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627515 5672 flags.go:64] FLAG: --max-pods="110" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627518 5672 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627522 5672 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627525 5672 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627528 5672 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627531 5672 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627534 5672 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627537 5672 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627544 5672 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627547 5672 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627550 5672 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627553 5672 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627556 5672 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627559 5672 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627563 5672 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627566 5672 flags.go:64] FLAG: --port="10250" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627570 5672 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627574 5672 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c69704dfa6bc1438" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627577 5672 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627580 5672 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627583 5672 flags.go:64] FLAG: --register-node="true" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627586 5672 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627590 5672 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627593 5672 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:53:26.635643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627596 5672 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627600 5672 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627604 5672 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627607 5672 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627611 5672 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627613 5672 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627616 5672 flags.go:64] FLAG: --runonce="false" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627619 5672 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627623 5672 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627626 5672 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627628 5672 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627631 5672 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627634 5672 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627638 5672 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627641 5672 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627644 5672 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627647 5672 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627650 5672 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627653 5672 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627656 5672 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627659 5672 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627662 5672 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627667 5672 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627670 5672 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627674 5672 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:53:26.636279 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627679 5672 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627682 5672 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627685 5672 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627688 5672 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627692 5672 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627695 5672 flags.go:64] FLAG: --v="2" Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627703 5672 flags.go:64] FLAG: --version="false" Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627707 5672 flags.go:64] FLAG: --vmodule="" Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627711 5672 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.627715 5672 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627846 5672 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627850 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627853 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627856 5672 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627860 5672 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627864 5672 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627868 5672 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627870 5672 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627873 5672 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627876 5672 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627879 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:53:26.636874 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627882 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627885 5672 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627888 5672 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627890 5672 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627893 5672 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627896 5672 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627899 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627901 5672 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627904 5672 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627907 5672 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627909 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627914 5672 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627917 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627919 5672 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627922 5672 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627925 5672 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627927 5672 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627930 5672 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627933 5672 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:53:26.637485 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627935 5672 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627938 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627941 5672 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627944 5672 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627946 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627949 5672 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627952 5672 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627955 5672 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627957 5672 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627960 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627962 5672 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627965 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627968 5672 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627970 5672 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627973 5672 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627976 5672 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627979 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627981 5672 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627984 5672 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627987 5672 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:53:26.637992 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627989 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627992 5672 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627995 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.627997 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628002 5672 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628005 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628008 5672 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628011 5672 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628014 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628016 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628019 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628022 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628025 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628027 5672 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628030 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628033 5672 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628036 5672 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628038 5672 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628041 5672 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:53:26.638598 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628045 5672 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628048 5672 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628051 5672 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628053 5672 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628056 5672 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628059 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628061 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628064 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628067 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628069 5672 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628072 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628075 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628078 5672 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628080 5672 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628083 5672 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628085 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628088 5672 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628093 5672 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628095 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628098 5672 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:53:26.639189 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628101 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628104 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628106 5672 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628109 5672 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628114 5672 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628117 5672 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628120 5672 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628123 5672 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628125 5672 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628129 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628131 5672 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628134 5672 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628138 5672 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628141 5672 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628158 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628162 5672 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:53:26.639760 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628165 5672 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.628193 5672 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628228 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628233 5672 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628236 5672 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628239 5672 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628242 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628245 5672 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628247 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628250 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628253 5672 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628256 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628258 5672 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628261 5672 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628265 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:53:26.640163 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628268 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628278 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628281 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628284 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628287 5672 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628290 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628292 5672 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628295 5672 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628297 5672 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628300 5672 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628303 5672 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628313 5672 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628316 5672 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628320 5672 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628323 5672 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628326 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628329 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628331 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628334 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628337 5672 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:53:26.640531 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628339 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628342 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628345 5672 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628347 5672 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628350 5672 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628353 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628355 5672 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628358 5672 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628361 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628363 5672 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628366 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628370 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628373 5672 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628376 5672 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628380 5672 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628383 5672 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628386 5672 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628388 5672 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628391 5672 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:53:26.641022 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628394 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628396 5672 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628399 5672 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628401 5672 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628404 5672 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628412 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628417 5672 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628420 5672 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628422 5672 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628425 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628427 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628430 5672 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628434 5672 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628438 5672 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628441 5672 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628444 5672 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628447 5672 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628450 5672 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628453 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628457 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:53:26.641578 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628459 5672 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628462 5672 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628465 5672 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628467 5672 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628471 5672 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628474 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628476 5672 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628479 5672 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628482 5672 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628485 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628487 5672 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628490 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628492 5672 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628495 5672 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628498 5672 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628500 5672 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628503 5672 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628506 5672 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628515 5672 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:53:26.642048 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628518 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628521 5672 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628524 5672 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628526 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628529 5672 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628532 5672 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628534 5672 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628537 5672 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628539 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628542 5672 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628544 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628547 5672 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628549 5672 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628552 5672 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.628554 5672 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.633413 5672 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.633430 5672 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.633455 5672 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.633460 5672 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:53:26.642535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633518 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633524 5672 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633529 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633533 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633536 5672 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633539 5672 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633542 5672 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633545 5672 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633548 5672 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633551 5672 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633553 5672 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633556 5672 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633560 5672 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633563 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633565 5672 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633569 5672 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633573 5672 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633576 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633579 5672 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:53:26.642981 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633582 5672 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633585 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633587 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633590 5672 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633593 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633596 5672 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633599 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633602 5672 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633605 5672 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633608 5672 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633611 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633614 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633616 5672 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633619 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633622 5672 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633624 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633627 5672 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633630 5672 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633632 5672 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633635 5672 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:53:26.643448 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633638 5672 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633641 5672 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633644 5672 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633647 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633649 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633652 5672 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633654 5672 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633657 5672 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633660 5672 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633663 5672 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633666 5672 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633668 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633671 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633674 5672 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633677 5672 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633679 5672 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633682 5672 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633684 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633687 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:53:26.643918 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633690 5672 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633693 5672 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633696 5672 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633699 5672 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633701 5672 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633704 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633707 5672 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633709 5672 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633712 5672 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633715 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633717 5672 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633720 5672 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633722 5672 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633725 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633728 5672 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633730 5672 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633733 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633735 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633738 5672 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633741 5672 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:53:26.644392 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633743 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633751 5672 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633754 5672 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633756 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633759 5672 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633762 5672 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633764 5672 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633767 5672 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633770 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633772 5672 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633775 5672 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633778 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633780 5672 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633784 5672 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633786 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633789 5672 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633791 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633794 5672 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633797 5672 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:53:26.644893 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633799 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633802 5672 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633805 5672 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633808 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633810 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633813 5672 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633816 5672 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633818 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633821 5672 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.633846 5672 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633978 5672 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633982 5672 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633986 5672 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633989 5672 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633991 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633995 5672 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:53:26.645569 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.633998 5672 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634001 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634004 5672 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634007 5672 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634009 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634012 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634015 5672 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634018 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634020 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634023 5672 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634026 5672 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634028 5672 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634031 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634034 5672 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634036 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634039 5672 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634042 5672 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634044 5672 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634047 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634050 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:53:26.646200 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634052 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634055 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634058 5672 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634060 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634063 5672 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634065 5672 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634069 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634071 5672 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634074 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634076 5672 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634079 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634082 5672 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634085 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634088 5672 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634092 5672 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634096 5672 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634099 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634102 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634105 5672 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:53:26.646731 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634108 5672 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634110 5672 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634113 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634115 5672 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634118 5672 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634121 5672 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634123 5672 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634126 5672 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634129 5672 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634132 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634135 5672 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634137 5672 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634140 5672 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634142 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634166 5672 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634169 5672 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634172 5672 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634175 5672 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634177 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634180 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:53:26.647259 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634183 5672 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634187 5672 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634190 5672 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634193 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634196 5672 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634198 5672 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634203 5672 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634205 5672 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634208 5672 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634210 5672 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634213 5672 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634216 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634218 5672 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634221 5672 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634223 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634226 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634229 5672 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634231 5672 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634234 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634237 5672 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:53:26.647743 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634239 5672 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634242 5672 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634244 5672 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634247 5672 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634249 5672 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634252 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634254 5672 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634257 5672 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634259 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634262 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634264 5672 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634267 5672 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634269 5672 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634272 5672 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634277 5672 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634280 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634282 5672 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634285 5672 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634295 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:53:26.648247 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634299 5672 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634301 5672 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.634326 5672 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634356 5672 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634360 5672 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634363 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634366 5672 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634368 5672 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634371 5672 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634374 5672 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634377 5672 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634379 5672 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634382 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634384 5672 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634387 5672 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634390 5672 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:53:26.648716 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634392 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634395 5672 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634397 5672 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634400 5672 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634403 5672 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634406 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634409 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634411 5672 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634415 5672 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634419 5672 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634422 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634426 5672 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634429 5672 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634433 5672 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634436 5672 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634438 5672 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634441 5672 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634444 5672 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634447 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634449 5672 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:53:26.649097 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634452 5672 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634454 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634457 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634460 5672 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634462 5672 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634465 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634467 5672 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634470 5672 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634472 5672 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634475 5672 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634478 5672 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634480 5672 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634484 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634486 5672 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634489 5672 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634492 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634495 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634497 5672 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634500 5672 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:53:26.649710 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634502 5672 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634505 5672 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634507 5672 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634510 5672 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634513 5672 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634516 5672 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634518 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634521 5672 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634524 5672 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634526 5672 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634529 5672 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634532 5672 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634535 5672 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634538 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634540 5672 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634543 5672 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634545 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634548 5672 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634551 5672 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:53:26.650179 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634553 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634555 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634558 5672 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634562 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634565 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634568 5672 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634570 5672 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634573 5672 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634576 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634578 5672 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634581 5672 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634583 5672 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634587 5672 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634589 5672 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634592 5672 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634595 5672 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634597 5672 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634600 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634603 5672 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:53:26.650639 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634606 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634608 5672 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634611 5672 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634613 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634616 5672 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634618 5672 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634621 5672 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634624 5672 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634626 5672 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634630 5672 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634632 5672 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634637 5672 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634641 5672 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634644 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634647 5672 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:53:26.634649 5672 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.634722 5672 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.635679 5672 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.635729 5672 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 22:53:26.651134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.635933 5672 server.go:1014] "Starting client certificate rotation" Nov 02 22:53:26.651656 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.636066 5672 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:53:26.651656 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.636119 5672 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:47:13 +0000 UTC" deadline="2028-07-04 08:36:39.962067612 +0000 UTC" Nov 02 22:53:26.651656 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.636155 5672 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14625h43m13.32592337s" Nov 02 22:53:26.651656 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.636378 5672 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:53:26.651656 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.637428 5672 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:53:26.651656 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.638665 5672 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:53:26.651656 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.640684 5672 log.go:25] "Validated CRI v1 image API" Nov 02 22:53:26.651656 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.641368 5672 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:53:26.651656 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.643638 5672 fs.go:135] Filesystem UUIDs: map[2604c32e-02e9-4bee-9f55-1a62368e91c4:/dev/nvme0n1p4 5bcfb583-c16d-44bd-a8e8-ed341fc3c4ef:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 02 22:53:26.651917 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.643658 5672 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00f022618f89226e2a89e07fdd6cd410aee7a2e86c1d088d79f20bfdfab4c937/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00f022618f89226e2a89e07fdd6cd410aee7a2e86c1d088d79f20bfdfab4c937/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06a3857c77677d72436a1c3f0a7956026adcef78140423cd5fb02694de51ec39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06a3857c77677d72436a1c3f0a7956026adcef78140423cd5fb02694de51ec39/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07fde3b76be6a14be4598fa33edaa6ecf159820c47afd3b0aea63a609037b676/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07fde3b76be6a14be4598fa33edaa6ecf159820c47afd3b0aea63a609037b676/userdata/shm major:0 minor:392 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1214330d59b32972f40ac043f7fe5926ff65fbdee121a7cd3a0f93b3bec8232e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1214330d59b32972f40ac043f7fe5926ff65fbdee121a7cd3a0f93b3bec8232e/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a76bd241011454f4ec6980e8eaad00d38048b00351566c3049ebc32a954c143/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a76bd241011454f4ec6980e8eaad00d38048b00351566c3049ebc32a954c143/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c4b324d7b7cf8c78a0eeb32bfedc7b0aa61951ecd900ffba76c224cad5d0da7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c4b324d7b7cf8c78a0eeb32bfedc7b0aa61951ecd900ffba76c224cad5d0da7/userdata/shm major:0 minor:429 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/270a07fcaa569ce34c85b7e36e5da97c8ef7ce2d616184266fc32999c494a74d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/270a07fcaa569ce34c85b7e36e5da97c8ef7ce2d616184266fc32999c494a74d/userdata/shm major:0 minor:434 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d5057cd8d40b089ab4e0e3c20ae840fe474f35468fcfd6149b68e80cde2d523/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d5057cd8d40b089ab4e0e3c20ae840fe474f35468fcfd6149b68e80cde2d523/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4325406e33abdacdd346295a9aa0af7461ef4610bdaeaa3aae5109b28ad93555/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4325406e33abdacdd346295a9aa0af7461ef4610bdaeaa3aae5109b28ad93555/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4981edabeec8a7f8151ba294e37df74d845c9e06642559be4ee3aacd6870b985/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4981edabeec8a7f8151ba294e37df74d845c9e06642559be4ee3aacd6870b985/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d634617a9a6f7d958a1241de51b5b4acedff197bdbe5fd0acf84bab0000d5d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d634617a9a6f7d958a1241de51b5b4acedff197bdbe5fd0acf84bab0000d5d5/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57800bd84f651ea3ff54b9079349ff3ce771887ec6fc9cb597e466a8b424371b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57800bd84f651ea3ff54b9079349ff3ce771887ec6fc9cb597e466a8b424371b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d3263b7b47ac08bcd8d88fe8109bf9403ecf5ca6adc100a11549c9a346ed4dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d3263b7b47ac08bcd8d88fe8109bf9403ecf5ca6adc100a11549c9a346ed4dc/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84552e920a03143762c5602455e2ccde8a705651a80b46572125592ac6840fec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84552e920a03143762c5602455e2ccde8a705651a80b46572125592ac6840fec/userdata/shm major:0 minor:432 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a4df370ea05876a66ed6da79c4b1f7e85906f89b6ffd07d4ba0b8ed637e490e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a4df370ea05876a66ed6da79c4b1f7e85906f89b6ffd07d4ba0b8ed637e490e/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a23e71108770675e264786c5b980bbea88535472d6caaf6a2a8ecfcbd8f178f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a23e71108770675e264786c5b980bbea88535472d6caaf6a2a8ecfcbd8f178f3/userdata/shm major:0 minor:268 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a48048e59f0de866ceb5f1c9e9ea470684afa28ea21911a4b5c325d513c55509/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a48048e59f0de866ceb5f1c9e9ea470684afa28ea21911a4b5c325d513c55509/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a79d7d3a32c515e2fc0fe618cc824ac64767af250e79f9f8f7f4ae8d94acda0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a79d7d3a32c515e2fc0fe618cc824ac64767af250e79f9f8f7f4ae8d94acda0b/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad78eb0d0ea0aca8fdc25a38d1e5b95999d27e52d347a4d33057a06a23f65abb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad78eb0d0ea0aca8fdc25a38d1e5b95999d27e52d347a4d33057a06a23f65abb/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1b885e08fa88c84d5b63e15a29e322c9289437ebc9ea417adeca61da9e9dc06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1b885e08fa88c84d5b63e15a29e322c9289437ebc9ea417adeca61da9e9dc06/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c39e16201ad053f0f82704ec6b09adaa194cc5ae5f191410f2a81b7961cbe360/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c39e16201ad053f0f82704ec6b09adaa194cc5ae5f191410f2a81b7961cbe360/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c613782942a74e0881d589b6c646cef4bdf2ca31db51a2230fd393fb48897d60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c613782942a74e0881d589b6c646cef4bdf2ca31db51a2230fd393fb48897d60/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/feb8301fde9e6bbef58794904ee0dfbe1e30443ec34482097009f23b2aaa902a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/feb8301fde9e6bbef58794904ee0dfbe1e30443ec34482097009f23b2aaa902a/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0798910d-3562-4dfe-b069-587510b73467/volumes/kubernetes.io~projected/kube-api-access-qrnjr:{mountpoint:/var/lib/kubelet/pods/0798910d-3562-4dfe-b069-587510b73467/volumes/kubernetes.io~projected/kube-api-access-qrnjr major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0798910d-3562-4dfe-b069-587510b73467/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/0798910d-3562-4dfe-b069-587510b73467/volumes/kubernetes.io~secret/metrics-certs major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a4f8b44-11be-4522-806a-87f1e54ba005/volumes/kubernetes.io~projected/kube-api-access-r6w4k:{mountpoint:/var/lib/kubelet/pods/2a4f8b44-11be-4522-806a-87f1e54ba005/volumes/kubernetes.io~projected/kube-api-access-r6w4k major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a4f8b44-11be-4522-806a-87f1e54ba005/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2a4f8b44-11be-4522-806a-87f1e54ba005/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41883b9a-e520-4e4d-8753-4512e53ff141/volumes/kubernetes.io~projected/kube-api-access-pqkgp:{mountpoint:/var/lib/kubelet/pods/41883b9a-e520-4e4d-8753-4512e53ff141/volumes/kubernetes.io~projected/kube-api-access-pqkgp major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41883b9a-e520-4e4d-8753-4512e53ff141/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/41883b9a-e520-4e4d-8753-4512e53ff141/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41883b9a-e520-4e4d-8753-4512e53ff141/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/41883b9a-e520-4e4d-8753-4512e53ff141/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~projected/kube-api-access-nqxx5:{mountpoint:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~projected/kube-api-access-nqxx5 major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/526ff9d4-f30b-4414-849b-c3e747329dd9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/526ff9d4-f30b-4414-849b-c3e747329dd9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/526ff9d4-f30b-4414-849b-c3e747329dd9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/526ff9d4-f30b-4414-849b-c3e747329dd9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/526ff9d4-f30b-4414-849b-c3e747329dd9/volumes/kubernetes.io~projected/kube-api-access-xtsgl:{mountpoint:/var/lib/kubelet/pods/526ff9d4-f30b-4414-849b-c3e747329dd9/volumes/kubernetes.io~projected/kube-api-access-xtsgl major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52e9c219-9e54-43c4-9dae-7d60c8e33dfe/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/52e9c219-9e54-43c4-9dae-7d60c8e33dfe/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526/volumes/kubernetes.io~projected/kube-api-access-jkstf:{mountpoint:/var/lib/kubelet/pods/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526/volumes/kubernetes.io~projected/kube-api-access-jkstf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/584a133a-3ca7-4a1b-88fc-83d98e3abc3d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/584a133a-3ca7-4a1b-88fc-83d98e3abc3d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/584a133a-3ca7-4a1b-88fc-83d98e3abc3d/volumes/kubernetes.io~projected/kube-api-access-9kbgs:{mountpoint:/var/lib/kubelet/pods/584a133a-3ca7-4a1b-88fc-83d98e3abc3d/volumes/kubernetes.io~projected/kube-api-access-9kbgs major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/584a133a-3ca7-4a1b-88fc-83d98e3abc3d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/584a133a-3ca7-4a1b-88fc-83d98e3abc3d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75cd1116-7b34-46e7-9502-fa71e12bbd9c/volumes/kubernetes.io~projected/kube-api-access-dtxmf:{mountpoint:/var/lib/kubelet/pods/75cd1116-7b34-46e7-9502-fa71e12bbd9c/volumes/kubernetes.io~projected/kube-api-access-dtxmf major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75cd1116-7b34-46e7-9502-fa71e12bbd9c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/75cd1116-7b34-46e7-9502-fa71e12bbd9c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d17af01-21f0-48b5-b6b3-834a2e2c7852/volumes/kubernetes.io~projected/kube-api-access-pqjjh:{mountpoint:/var/lib/kubelet/pods/7d17af01-21f0-48b5-b6b3-834a2e2c7852/volumes/kubernetes.io~projected/kube-api-access-pqjjh major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d17af01-21f0-48b5-b6b3-834a2e2c7852/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7d17af01-21f0-48b5-b6b3-834a2e2c7852/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d17af01-21f0-48b5-b6b3-834a2e2c7852/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/7d17af01-21f0-48b5-b6b3-834a2e2c7852/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/889d0ebb-164e-4c18-973a-509f4dcfa1bc/volumes/kubernetes.io~projected/kube-api-access-tlv6j:{mountpoint:/var/lib/kubelet/pods/889d0ebb-164e-4c18-973a-509f4dcfa1bc/volumes/kubernetes.io~projected/kube-api-access-tlv6j major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78/volumes/kubernetes.io~projected/kube-api-access-gkvpl:{mountpoint:/var/lib/kubelet/pods/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78/volumes/kubernetes.io~projected/kube-api-access-gkvpl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad49b548-e79b-43aa-9479-57c1e6d06fdd/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ad49b548-e79b-43aa-9479-57c1e6d06fdd/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae18bf8e-ca0b-41e8-90ca-9e830160012a/volumes/kubernetes.io~projected/kube-api-access-jtz2w:{mountpoint:/var/lib/kubelet/pods/ae18bf8e-ca0b-41e8-90ca-9e830160012a/volumes/kubernetes.io~projected/kube-api-access-jtz2w major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aeea33da-f0ef-4912-8ebc-29b48d336448/volumes/kubernetes.io~projected/kube-api-access-qlrjz:{mountpoint:/var/lib/kubelet/pods/aeea33da-f0ef-4912-8ebc-29b48d336448/volumes/kubernetes.io~projected/kube-api-access-qlrjz major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aeea33da-f0ef-4912-8ebc-29b48d336448/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/aeea33da-f0ef-4912-8ebc-29b48d336448/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aeea33da-f0ef-4912-8ebc-29b48d336448/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/aeea33da-f0ef-4912-8ebc-29b48d336448/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b37eb397-e28e-49c2-a711-a4efd3466679/volumes/kubernetes.io~projected/kube-api-access-7pcfv:{mountpoint:/var/lib/kubelet/pods/b37eb397-e28e-49c2-a711-a4efd3466679/volumes/kubernetes.io~projected/kube-api-access-7pcfv major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8238d30-1c60-4c49-9171-881a3f46f399/volumes/kubernetes.io~projected/kube-api-access-q7mrv:{mountpoint:/var/lib/kubelet/pods/b8238d30-1c60-4c49-9171-881a3f46f399/volumes/kubernetes.io~projected/kube-api-access-q7mrv major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8238d30-1c60-4c49-9171-881a3f46f399/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/b8238d30-1c60-4c49-9171-881a3f46f399/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8ae4d29-9676-4fff-a0cb-a26179a61e50/volumes/kubernetes.io~projected/kube-api-access-hcmgd:{mountpoint:/var/lib/kubelet/pods/c8ae4d29-9676-4fff-a0cb-a26179a61e50/volumes/kubernetes.io~projected/kube-api-access-hcmgd major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce280d50-8be5-4223-91ce-3ea1da4e7ff6/volumes/kubernetes.io~projected/kube-api-access-97g84:{mountpoint:/var/lib/kubelet/pods/ce280d50-8be5-4223-91ce-3ea1da4e7ff6/volumes/kubernetes.io~projected/kube-api-access-97g84 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e26c39e8-2ffc-4179-8498-76167f635e9a/volumes/kubernetes.io~projected/kube-api-access-4pnv7:{mountpoint:/var/lib/kubelet/pods/e26c39e8-2ffc-4179-8498-76167f635e9a/volumes/kubernetes.io~projected/kube-api-access-4pnv7 major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e26c39e8-2ffc-4179-8498-76167f635e9a/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/e26c39e8-2ffc-4179-8498-76167f635e9a/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volume-subpaths/web-config/config-reloader/2 major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volume-subpaths/web-config/prometheus/15 major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~empty-dir/config-out major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~projected/kube-api-access-qbf65:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~projected/kube-api-access-qbf65 major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~projected/tls-assets major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/config major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:389 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/web-config major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe3683b5-cfe1-408e-be5b-5a81c36304c4/volumes/kubernetes.io~projected/kube-api-access-4f2j4:{mountpoint:/var/lib/kubelet/pods/fe3683b5-cfe1-408e-be5b-5a81c36304c4/volumes/kubernetes.io~projected/kube-api-access-4f2j4 major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe3683b5-cfe1-408e-be5b-5a81c36304c4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fe3683b5-cfe1-408e-be5b-5a81c36304c4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe3683b5-cfe1-408e-be5b-5a81c36304c4/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/fe3683b5-cfe1-408e-be5b-5a81c36304c4/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:263 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/8b63932ea895c88b8884a83a834b5b27a1951e309c6804c35436e119daba9b86/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f0cdc7d3c88d5efc381616a2bfcdfa78d6b10932cb869c898db324b2b37d305a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c8bad4921fdfde4d23fbd5964e4e9ea4247c70fd0a29b16150e449161cd397a2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/f0544f8043030d7b1ebfccc1108f627e0bc95918bd7b703cf213a968d4029ff1/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/af7aa02c2919cfdc6139ec1b02440924d2fda9bb289851d14202b6b1baf90997/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/632b32b129f925eefc1118268ddcc6fa42f9ebea7fbe57f46e98f6f5ce9f0c84/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/30988211b3da5e23f0b3d0b0768fc036daee1f8e44cba417c34d03aa75e602de/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/76085413925f15ec8bfc12e05de017d26c091fbbacf3d4091c82ae8e51ee9311/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/eb88bf21d595669f6dabdb84801f65062e41007b0f3899021a139c6f73c2eebb/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b71b11c96a7f24c8dfa8054205eba235eb57206326ad2967288681bfdbecbcc6/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/c5883510cfbe418ef87d34d0a9a432dee182357210eab5e863fb72f9f1ee8918/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/9bccb95d2232c76f6859bfd407254f6ff5f24f13cb0248d144eb9145e66579d0/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/705e5efc6e429e6de4067d68579719ffc34577255d955aaba3294a2cfa9a507d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/99309071e01b9b08344b9923551617e68c7c62edb7b57c89beefa3377cade892/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/570001127ee1794581c7f84c8f8da200c3b8cd9c0f855c6fb47efd26a3076a1e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/64bf6ba4c6b0acfd697238c3637259afc4d991e471f70bcbffc08a079c9d6d10/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/53c5998b56877679703db6b1614c1469e84a4b918e89cf18cb2adf18d06690b2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/64bab9ed9159c3997d75e7da66303d47760fa890ecb9981accc607cd6ede8d85/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c74311dd45d9c91c5b80708933135df147dfa6856f0982c1b463bdbe8eb6baff/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/aaee8c14c058ba8fff18415e3f41ed3f7fcab0adbaaa241908bdf65cdfa7a01f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/10fc6b4be3742be9f8b98ea26292fd00f5a9737ed2c53d5806e8343526d0e7eb/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/2193b7118aa4e6777db1e91d5c6e66c74653f8e535aa48a035c9aede5f26b2f5/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/c13ed789818238d7188546cd6ca02c04842673ff9f68c69a783a5210de8f8bb7/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/da87beeeed960270fecbf98199011861704a4bed95be0e7ebf2e0bc73bb2f862/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/897f791751c47b396783c87d596ca427126c88767d6e3cb2aa12e0876ee47989/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/e6d4a3bea18b43a6b40fa0c47858c65528af9aa56943efe46b639bfbe11c061d/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/24a5cc20caa5e8c9675571223c4ff35cfe4fb89c2ecbeb1227c1dcc174633e7c/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/3b5f151bc7353981365d3b8c74a4df05703a97f399490d0171cab69a8e620b1c/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/220a295f16dee8e5f8cd44ed2d48ce8581f1827098c179c6fbbb47d5cd091d60/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/4b40b747d0232c9eebbb7fadb63c6379362189a20faca1cc093e809494c8db58/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/abc1d6c859252176564de67b9fbb39aa8f4d9228bc4628c70fa42b74c2dcbfe0/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/03b0f975e83153d968a65e40c6f57d24201a99abba7e2b8d778a79e5dbb4cbe1/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/b380d63652e3831b021895be10f3322da2e72181a71703a62da7fb9405bf93f6/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/c8839d3e423079ab35ff1bbe841cac327e9a1a51c44798615e95dcdaf1271332/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/f5859479235f399a0781dfe58aa72da7a28bd986849ab8e1a70726d874abdc31/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/57e1b4cece6db5abceb2e6f95cda1d08cd16f4cae30720cd66c2e76b627983c7/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/76bcbd9d7201d557ef9f8aefdd142e622d50f48bba41d4663b9bbab4231ca3f1/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/6bd1ae39da806528f8f79fef80897c96a32fed154b7b32e0cb9f9a76142851a9/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/10cd1dc88c2c5e1e41742de00e3daf295cb51d4a97f26549f201bb9095875d5e/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/22f1564b3d42b3d710ee1bf700153253074915ab8e4c42a4511661ed28111415/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/566fc353ae39df4d15d5d08419a174fa98468dc3fa912adf5c61391ff4d1ae0b/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/e48f4054dde01826578258c6a0744a5c8744047a7a848e4ee13203b4629d922f/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/37c56a1f139c8aa182b265660db55bb326e1e144c13b66703ce4d64780af7a4c/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/f927fe12f85d2a9ebebeca97228ac650559fe890060982c0095106c9455ed5b9/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/76c0693e69209212cd1ffa91d55af9a33dfcc9befa3027ffc6742216556ce206/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/6745ab47ca1e054a12442e4c4ae821f135407601255ebaf777c6958e5d1f374e/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/5de5b495adba86bda20bfa87fc3273f4dc2316c113505866e98a6a9b89075c7f/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/fb2e9e046713c4399b0bebe6f2facc08ce4759b6601bdca51e67165ee53387be/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/0953852f0b294bbcd449d24098b95d1574ac4291a72fa4d468cce382b5b5a591/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/91f76f2884203e122c54d36ca7c27e02667c9c83dbbdd992cde38490f01bd2d5/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/72eeb5d4ab97212da2d3a8df1d363215076e3070e1e8237941c95328cde277a5/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/d4aa39d97950518afe916c91179fac3b5080e8d4d562f8143a137af4d804fd25/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/c9835f90d98d01a1bc5ebe3a71a83f55be7c29c1394ec36ad193ca8d1ed50aa9/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/924f454e07256aca2848e1ce86fa390271019b7e689933d10c51931a3012227e/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/d244c387f88341f9c3dc791987ef1bdc26ae36b20c407aace3e25df739de1946/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/c58a3c8af423f928b1552a69716ae7d0fc0233cb62b1f98682c81aec71d5a6c1/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/e7f424f4f96a80bf6df4bb8627183f7404fe0e09da2a3a284e6ca08dff3d5db5/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/2c3c626394af84305e3c1ce507c129d0e094ce6507f9cb9b023b130d53c05835/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/eb149499c2f0dea64232e0689150baf1aa7ca39173b06d9b9db3bb4e1ee62d39/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/43e2e6c66160307dee49d39320a06feb8ad3c588149e0b4025bc21720c0578a3/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/fec7d8758890317df58afaec8e1f6087ce2cca006835b045f75a6218525b86ac/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/fc273c1bc1c29e9c54d52ab04142ce1dfe9ffab5a0fe3aeff150bb38f44f9869/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/1ba08345b3297187f63590f9888a728d0ec4af310d9325385f8f5520e8f99f4f/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/b6877e061b5e924724cc32c9d3b32e23024398ce0a0578810c2b6ab1ac1fc647/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/570c37b6ebce48fbb95cf1c0abf64b33dee19f5f769fa73505e04e6930cf98a6/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/8da44b8cb60843ac999e11d87ef88fe30919f18a44a3ba831f6a19e05ac4ab3c/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/e618e48d167b728a66bfd861852f8937692881532b8279771a11df13dc12c096/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/54d397d56af1350dd58168761e88bdb1ed58be415750b054ae99ec1e89460aa2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/efab8dcc2b662e0a42b19e71e1a4c616380a410ddb374f202764568bb9674714/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/112a0d1f2610770efc7e20fa2dc60507e6a8846b5ce55458e30e225a860c45e0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/a8c99534d23ee3576f120d9fb24d7c80f706f6bd6a15dfe556545987aa284038/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/dc25bd60c46ca93ad70a77987871fa724dd254a07b19376e41dff83af59326e6/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/00693c437e36ae41cce3d23693f6dafb5f9da8037c695e67fc229ce6316c1342/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/453686baec04225e069570cbc6c04e14b1c782309d86c04a5477f2fff1382356/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/272b9c772541ecdfad929025b4fb86e674bfd98b41a365e073b5281ea9388ada/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/676c03dfdc3558f78febdeab6e26278b63af742e30e9e51b90225305ff3ae610/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 22:53:26.653164 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.652729 5672 manager.go:217] Machine: {Timestamp:2026-11-02 22:53:26.652259501 +0000 UTC m=+0.059571322 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103505 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a6ce6cbc56d06a867715f9b0a6994 SystemUUID:ec2a6ce6-cbc5-6d06-a867-715f9b0a6994 BootID:13a6952a-1544-4755-bfdc-8dc93267713e Filesystems:[{Device:/var/lib/kubelet/pods/41883b9a-e520-4e4d-8753-4512e53ff141/volumes/kubernetes.io~projected/kube-api-access-pqkgp DeviceMajor:0 DeviceMinor:267 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c8ae4d29-9676-4fff-a0cb-a26179a61e50/volumes/kubernetes.io~projected/kube-api-access-hcmgd DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a79d7d3a32c515e2fc0fe618cc824ac64767af250e79f9f8f7f4ae8d94acda0b/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e26c39e8-2ffc-4179-8498-76167f635e9a/volumes/kubernetes.io~projected/kube-api-access-4pnv7 DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:389 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c4b324d7b7cf8c78a0eeb32bfedc7b0aa61951ecd900ffba76c224cad5d0da7/userdata/shm DeviceMajor:0 DeviceMinor:429 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/526ff9d4-f30b-4414-849b-c3e747329dd9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad49b548-e79b-43aa-9479-57c1e6d06fdd/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:428 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0798910d-3562-4dfe-b069-587510b73467/volumes/kubernetes.io~projected/kube-api-access-qrnjr DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae18bf8e-ca0b-41e8-90ca-9e830160012a/volumes/kubernetes.io~projected/kube-api-access-jtz2w DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41883b9a-e520-4e4d-8753-4512e53ff141/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:261 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a4f8b44-11be-4522-806a-87f1e54ba005/volumes/kubernetes.io~projected/kube-api-access-r6w4k DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4325406e33abdacdd346295a9aa0af7461ef4610bdaeaa3aae5109b28ad93555/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75cd1116-7b34-46e7-9502-fa71e12bbd9c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57800bd84f651ea3ff54b9079349ff3ce771887ec6fc9cb597e466a8b424371b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/584a133a-3ca7-4a1b-88fc-83d98e3abc3d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/7d17af01-21f0-48b5-b6b3-834a2e2c7852/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e26c39e8-2ffc-4179-8498-76167f635e9a/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78/volumes/kubernetes.io~projected/kube-api-access-gkvpl DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/526ff9d4-f30b-4414-849b-c3e747329dd9/volumes/kubernetes.io~projected/kube-api-access-xtsgl DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2a4f8b44-11be-4522-806a-87f1e54ba005/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75cd1116-7b34-46e7-9502-fa71e12bbd9c/volumes/kubernetes.io~projected/kube-api-access-dtxmf DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aeea33da-f0ef-4912-8ebc-29b48d336448/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a76bd241011454f4ec6980e8eaad00d38048b00351566c3049ebc32a954c143/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41883b9a-e520-4e4d-8753-4512e53ff141/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:264 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:384 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d5057cd8d40b089ab4e0e3c20ae840fe474f35468fcfd6149b68e80cde2d523/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/584a133a-3ca7-4a1b-88fc-83d98e3abc3d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:390 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/270a07fcaa569ce34c85b7e36e5da97c8ef7ce2d616184266fc32999c494a74d/userdata/shm DeviceMajor:0 DeviceMinor:434 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce280d50-8be5-4223-91ce-3ea1da4e7ff6/volumes/kubernetes.io~projected/kube-api-access-97g84 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe3683b5-cfe1-408e-be5b-5a81c36304c4/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:263 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~projected/kube-api-access-nqxx5 DeviceMajor:0 DeviceMinor:323 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:381 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84552e920a03143762c5602455e2ccde8a705651a80b46572125592ac6840fec/userdata/shm DeviceMajor:0 DeviceMinor:432 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad78eb0d0ea0aca8fdc25a38d1e5b95999d27e52d347a4d33057a06a23f65abb/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~projected/kube-api-access-qbf65 DeviceMajor:0 DeviceMinor:391 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d634617a9a6f7d958a1241de51b5b4acedff197bdbe5fd0acf84bab0000d5d5/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aeea33da-f0ef-4912-8ebc-29b48d336448/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:387 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d17af01-21f0-48b5-b6b3-834a2e2c7852/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:248 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:383 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:386 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:388 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:385 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d3263b7b47ac08bcd8d88fe8109bf9403ecf5ca6adc100a11549c9a346ed4dc/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/889d0ebb-164e-4c18-973a-509f4dcfa1bc/volumes/kubernetes.io~projected/kube-api-access-tlv6j DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/52e9c219-9e54-43c4-9dae-7d60c8e33dfe/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/526ff9d4-f30b-4414-849b-c3e747329dd9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8238d30-1c60-4c49-9171-881a3f46f399/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a23e71108770675e264786c5b980bbea88535472d6caaf6a2a8ecfcbd8f178f3/userdata/shm DeviceMajor:0 DeviceMinor:268 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:307 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aeea33da-f0ef-4912-8ebc-29b48d336448/volumes/kubernetes.io~projected/kube-api-access-qlrjz DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/feb8301fde9e6bbef58794904ee0dfbe1e30443ec34482097009f23b2aaa902a/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00f022618f89226e2a89e07fdd6cd410aee7a2e86c1d088d79f20bfdfab4c937/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1b885e08fa88c84d5b63e15a29e322c9289437ebc9ea417adeca61da9e9dc06/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d17af01-21f0-48b5-b6b3-834a2e2c7852/volumes/kubernetes.io~projected/kube-api-access-pqjjh DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c613782942a74e0881d589b6c646cef4bdf2ca31db51a2230fd393fb48897d60/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c39e16201ad053f0f82704ec6b09adaa194cc5ae5f191410f2a81b7961cbe360/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:377 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06a3857c77677d72436a1c3f0a7956026adcef78140423cd5fb02694de51ec39/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07fde3b76be6a14be4598fa33edaa6ecf159820c47afd3b0aea63a609037b676/userdata/shm DeviceMajor:0 DeviceMinor:392 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b8238d30-1c60-4c49-9171-881a3f46f399/volumes/kubernetes.io~projected/kube-api-access-q7mrv DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b37eb397-e28e-49c2-a711-a4efd3466679/volumes/kubernetes.io~projected/kube-api-access-7pcfv DeviceMajor:0 DeviceMinor:431 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526/volumes/kubernetes.io~projected/kube-api-access-jkstf DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a48048e59f0de866ceb5f1c9e9ea470684afa28ea21911a4b5c325d513c55509/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0798910d-3562-4dfe-b069-587510b73467/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:424 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4981edabeec8a7f8151ba294e37df74d845c9e06642559be4ee3aacd6870b985/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a4df370ea05876a66ed6da79c4b1f7e85906f89b6ffd07d4ba0b8ed637e490e/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1214330d59b32972f40ac043f7fe5926ff65fbdee121a7cd3a0f93b3bec8232e/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe3683b5-cfe1-408e-be5b-5a81c36304c4/volumes/kubernetes.io~projected/kube-api-access-4f2j4 DeviceMajor:0 DeviceMinor:266 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d158803-71d8-447b-8e4d-e73287682505/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/584a133a-3ca7-4a1b-88fc-83d98e3abc3d/volumes/kubernetes.io~projected/kube-api-access-9kbgs DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe3683b5-cfe1-408e-be5b-5a81c36304c4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:07fde3b76be6a14 MacAddress:0a:22:5c:f6:21:6e Speed:10000 Mtu:8901} {Name:1214330d59b3297 MacAddress:be:96:48:e7:6a:b4 Speed:10000 Mtu:8901} {Name:1a76bd241011454 MacAddress:3a:5c:7a:76:94:bc Speed:10000 Mtu:8901} {Name:1c4b324d7b7cf8c MacAddress:36:a0:51:38:2e:c9 Speed:10000 Mtu:8901} {Name:270a07fcaa569ce MacAddress:f2:95:de:a0:8a:e4 Speed:10000 Mtu:8901} {Name:84552e920a03143 MacAddress:a6:a6:67:b7:f6:77 Speed:10000 Mtu:8901} {Name:9a4df370ea05876 MacAddress:0a:46:28:33:fb:f1 Speed:10000 Mtu:8901} {Name:a23e71108770675 MacAddress:36:ff:1a:a7:a0:bb Speed:10000 Mtu:8901} {Name:a79d7d3a32c515e MacAddress:82:3f:c6:5c:d2:97 Speed:10000 Mtu:8901} {Name:ad78eb0d0ea0aca MacAddress:6e:66:73:f1:70:55 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f6:44:2f:af Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:45:aa:01:a5:41 Speed:0 Mtu:8901} {Name:c613782942a74e0 MacAddress:f6:cd:a1:d9:e1:44 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f6:44:2f:af Speed:0 Mtu:9001} {Name:feb8301fde9e6bb MacAddress:96:cd:a4:81:de:44 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:82:7b:61:2a:c8:7e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a2:36:34:0e:a1:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:53:26.653392 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.653168 5672 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:53:26.653392 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.653211 5672 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:53:26.653590 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.653567 5672 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:53:26.653720 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.653592 5672 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-4.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:53:26.653763 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.653723 5672 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:53:26.653763 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.653731 5672 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:53:26.653763 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.653742 5672 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:53:26.653763 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.653761 5672 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:53:26.653863 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.653848 5672 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:53:26.654029 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.654020 5672 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:53:26.654157 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.654137 5672 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:53:26.654194 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.654186 5672 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:53:26.654222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.654198 5672 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:53:26.654222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.654214 5672 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:53:26.654222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.654222 5672 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:53:26.654319 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.654230 5672 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:53:26.655182 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.655168 5672 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:53:26.655182 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.655181 5672 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:53:26.655663 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.655641 5672 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:53:26.658594 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.658577 5672 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:53:26.658594 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.658590 5672 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:53:26.658705 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.658681 5672 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 22:53:26.658860 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.658849 5672 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:53:26.658902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.658878 5672 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:53:26.658969 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.658960 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:53:26.659009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.658975 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:53:26.659009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.658981 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:53:26.659009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.658986 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:53:26.659009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.658992 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:53:26.659009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.658999 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:53:26.659009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.659005 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:53:26.659009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.659011 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:53:26.659303 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.659018 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:53:26.659303 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.659023 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:53:26.659303 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.659031 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:53:26.659303 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.659040 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:53:26.659303 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.659063 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:53:26.659303 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.659069 5672 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:53:26.662379 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.662367 5672 server.go:1263] "Started kubelet" Nov 02 22:53:26.663136 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.662459 5672 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:53:26.663136 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.662535 5672 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:53:26.663136 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.663046 5672 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:53:26.663034 ip-10-0-137-4 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:53:26.665493 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.665462 5672 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:53:26.666590 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.666575 5672 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:53:26.671749 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.671732 5672 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:53:26.671848 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.671829 5672 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:53:26.671917 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.671894 5672 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:47:15 +0000 UTC" deadline="2028-04-19 20:12:56.395391209 +0000 UTC" Nov 02 22:53:26.671965 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.671916 5672 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12813h19m29.723477828s" Nov 02 22:53:26.671965 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:26.671931 5672 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:53:26.672024 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.671980 5672 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:53:26.672024 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.671987 5672 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:53:26.672328 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.672310 5672 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:53:26.673047 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.673028 5672 factory.go:55] Registering systemd factory Nov 02 22:53:26.673137 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.673050 5672 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:53:26.673574 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.673558 5672 factory.go:153] Registering CRI-O factory Nov 02 22:53:26.673574 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.673575 5672 factory.go:223] Registration of the crio container factory successfully Nov 02 22:53:26.673702 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.673627 5672 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:53:26.673702 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.673649 5672 factory.go:103] Registering Raw factory Nov 02 22:53:26.673702 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.673679 5672 manager.go:1199] Started watching for new ooms in manager Nov 02 22:53:26.673962 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.673945 5672 manager.go:319] Starting recovery of all containers Nov 02 22:53:26.675082 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675035 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75cd1116-7b34-46e7-9502-fa71e12bbd9c" volumeName="kubernetes.io/secret/75cd1116-7b34-46e7-9502-fa71e12bbd9c-metrics-tls" seLinuxMountContext="" Nov 02 22:53:26.675224 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675083 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8238d30-1c60-4c49-9171-881a3f46f399" volumeName="kubernetes.io/projected/b8238d30-1c60-4c49-9171-881a3f46f399-kube-api-access-q7mrv" seLinuxMountContext="" Nov 02 22:53:26.675224 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675107 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 02 22:53:26.675224 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675121 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe3683b5-cfe1-408e-be5b-5a81c36304c4" volumeName="kubernetes.io/projected/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-api-access-4f2j4" seLinuxMountContext="" Nov 02 22:53:26.675224 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675135 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="526ff9d4-f30b-4414-849b-c3e747329dd9" volumeName="kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-tuned" seLinuxMountContext="" Nov 02 22:53:26.675224 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675171 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="889d0ebb-164e-4c18-973a-509f4dcfa1bc" volumeName="kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-daemon-config" seLinuxMountContext="" Nov 02 22:53:26.675224 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675184 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce280d50-8be5-4223-91ce-3ea1da4e7ff6" volumeName="kubernetes.io/projected/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kube-api-access-97g84" seLinuxMountContext="" Nov 02 22:53:26.675224 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675196 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-metrics-client-ca" seLinuxMountContext="" Nov 02 22:53:26.675224 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675211 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-kube-api-access-qbf65" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675233 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78" volumeName="kubernetes.io/configmap/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-serviceca" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675257 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad49b548-e79b-43aa-9479-57c1e6d06fdd" volumeName="kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675271 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8238d30-1c60-4c49-9171-881a3f46f399" volumeName="kubernetes.io/configmap/b8238d30-1c60-4c49-9171-881a3f46f399-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675284 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675296 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0798910d-3562-4dfe-b069-587510b73467" volumeName="kubernetes.io/projected/0798910d-3562-4dfe-b069-587510b73467-kube-api-access-qrnjr" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675313 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a4f8b44-11be-4522-806a-87f1e54ba005" volumeName="kubernetes.io/projected/2a4f8b44-11be-4522-806a-87f1e54ba005-kube-api-access-r6w4k" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675328 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41883b9a-e520-4e4d-8753-4512e53ff141" volumeName="kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-metrics-client-ca" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675340 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d17af01-21f0-48b5-b6b3-834a2e2c7852" volumeName="kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675380 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8ae4d29-9676-4fff-a0cb-a26179a61e50" volumeName="kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675392 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675405 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-web-config" seLinuxMountContext="" Nov 02 22:53:26.675439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675432 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="584a133a-3ca7-4a1b-88fc-83d98e3abc3d" volumeName="kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-config" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675453 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="584a133a-3ca7-4a1b-88fc-83d98e3abc3d" volumeName="kubernetes.io/secret/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675467 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d17af01-21f0-48b5-b6b3-834a2e2c7852" volumeName="kubernetes.io/configmap/7d17af01-21f0-48b5-b6b3-834a2e2c7852-metrics-client-ca" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675480 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae18bf8e-ca0b-41e8-90ca-9e830160012a" volumeName="kubernetes.io/projected/ae18bf8e-ca0b-41e8-90ca-9e830160012a-kube-api-access-jtz2w" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675491 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675512 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41883b9a-e520-4e4d-8753-4512e53ff141" volumeName="kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675525 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d158803-71d8-447b-8e4d-e73287682505" volumeName="kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675538 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52e9c219-9e54-43c4-9dae-7d60c8e33dfe" volumeName="kubernetes.io/configmap/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-konnectivity-ca" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675556 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8ae4d29-9676-4fff-a0cb-a26179a61e50" volumeName="kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-binary-copy" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675568 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8ae4d29-9676-4fff-a0cb-a26179a61e50" volumeName="kubernetes.io/projected/c8ae4d29-9676-4fff-a0cb-a26179a61e50-kube-api-access-hcmgd" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675580 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675592 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe3683b5-cfe1-408e-be5b-5a81c36304c4" volumeName="kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675604 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d158803-71d8-447b-8e4d-e73287682505" volumeName="kubernetes.io/configmap/4d158803-71d8-447b-8e4d-e73287682505-metrics-client-ca" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675617 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="526ff9d4-f30b-4414-849b-c3e747329dd9" volumeName="kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-tmp" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675641 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75cd1116-7b34-46e7-9502-fa71e12bbd9c" volumeName="kubernetes.io/projected/75cd1116-7b34-46e7-9502-fa71e12bbd9c-kube-api-access-dtxmf" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675654 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae18bf8e-ca0b-41e8-90ca-9e830160012a" volumeName="kubernetes.io/empty-dir/ae18bf8e-ca0b-41e8-90ca-9e830160012a-tmp-dir" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675666 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aeea33da-f0ef-4912-8ebc-29b48d336448" volumeName="kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675677 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aeea33da-f0ef-4912-8ebc-29b48d336448" volumeName="kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675690 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b37eb397-e28e-49c2-a711-a4efd3466679" volumeName="kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675702 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e26c39e8-2ffc-4179-8498-76167f635e9a" volumeName="kubernetes.io/secret/e26c39e8-2ffc-4179-8498-76167f635e9a-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675714 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41883b9a-e520-4e4d-8753-4512e53ff141" volumeName="kubernetes.io/projected/41883b9a-e520-4e4d-8753-4512e53ff141-kube-api-access-pqkgp" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675731 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526" volumeName="kubernetes.io/configmap/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-iptables-alerter-script" seLinuxMountContext="" Nov 02 22:53:26.675748 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675744 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="584a133a-3ca7-4a1b-88fc-83d98e3abc3d" volumeName="kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-script-lib" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675756 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="889d0ebb-164e-4c18-973a-509f4dcfa1bc" volumeName="kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cni-binary-copy" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675768 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-db" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675780 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675792 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0798910d-3562-4dfe-b069-587510b73467" volumeName="kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675815 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75cd1116-7b34-46e7-9502-fa71e12bbd9c" volumeName="kubernetes.io/empty-dir/75cd1116-7b34-46e7-9502-fa71e12bbd9c-tmp-dir" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675843 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d17af01-21f0-48b5-b6b3-834a2e2c7852" volumeName="kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675858 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="889d0ebb-164e-4c18-973a-509f4dcfa1bc" volumeName="kubernetes.io/projected/889d0ebb-164e-4c18-973a-509f4dcfa1bc-kube-api-access-tlv6j" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675878 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675890 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe3683b5-cfe1-408e-be5b-5a81c36304c4" volumeName="kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675902 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe3683b5-cfe1-408e-be5b-5a81c36304c4" volumeName="kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675925 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aeea33da-f0ef-4912-8ebc-29b48d336448" volumeName="kubernetes.io/empty-dir/aeea33da-f0ef-4912-8ebc-29b48d336448-tmp" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675937 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675949 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d158803-71d8-447b-8e4d-e73287682505" volumeName="kubernetes.io/empty-dir/4d158803-71d8-447b-8e4d-e73287682505-tmp" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675970 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75cd1116-7b34-46e7-9502-fa71e12bbd9c" volumeName="kubernetes.io/configmap/75cd1116-7b34-46e7-9502-fa71e12bbd9c-config-volume" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675982 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.675994 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-tls-assets" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676007 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41883b9a-e520-4e4d-8753-4512e53ff141" volumeName="kubernetes.io/empty-dir/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-textfile" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676019 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52e9c219-9e54-43c4-9dae-7d60c8e33dfe" volumeName="kubernetes.io/secret/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-agent-certs" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676032 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8238d30-1c60-4c49-9171-881a3f46f399" volumeName="kubernetes.io/empty-dir/b8238d30-1c60-4c49-9171-881a3f46f399-data-volume" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676044 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e26c39e8-2ffc-4179-8498-76167f635e9a" volumeName="kubernetes.io/projected/e26c39e8-2ffc-4179-8498-76167f635e9a-kube-api-access-4pnv7" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676056 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aeea33da-f0ef-4912-8ebc-29b48d336448" volumeName="kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676068 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aeea33da-f0ef-4912-8ebc-29b48d336448" volumeName="kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676082 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aeea33da-f0ef-4912-8ebc-29b48d336448" volumeName="kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676094 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d158803-71d8-447b-8e4d-e73287682505" volumeName="kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676107 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="526ff9d4-f30b-4414-849b-c3e747329dd9" volumeName="kubernetes.io/projected/526ff9d4-f30b-4414-849b-c3e747329dd9-kube-api-access-xtsgl" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676118 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78" volumeName="kubernetes.io/projected/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-kube-api-access-gkvpl" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676130 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8238d30-1c60-4c49-9171-881a3f46f399" volumeName="kubernetes.io/secret/b8238d30-1c60-4c49-9171-881a3f46f399-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676141 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-config" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676169 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-metrics-client-certs" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676182 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41883b9a-e520-4e4d-8753-4512e53ff141" volumeName="kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676195 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d158803-71d8-447b-8e4d-e73287682505" volumeName="kubernetes.io/projected/4d158803-71d8-447b-8e4d-e73287682505-kube-api-access-nqxx5" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676207 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d158803-71d8-447b-8e4d-e73287682505" volumeName="kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676220 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="584a133a-3ca7-4a1b-88fc-83d98e3abc3d" volumeName="kubernetes.io/projected/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-kube-api-access-9kbgs" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676235 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aeea33da-f0ef-4912-8ebc-29b48d336448" volumeName="kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676249 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8ae4d29-9676-4fff-a0cb-a26179a61e50" volumeName="kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676262 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676275 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-config-out" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676287 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d158803-71d8-447b-8e4d-e73287682505" volumeName="kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-tls" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676301 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d158803-71d8-447b-8e4d-e73287682505" volumeName="kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676316 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="584a133a-3ca7-4a1b-88fc-83d98e3abc3d" volumeName="kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-env-overrides" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676329 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d17af01-21f0-48b5-b6b3-834a2e2c7852" volumeName="kubernetes.io/projected/7d17af01-21f0-48b5-b6b3-834a2e2c7852-kube-api-access-pqjjh" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676342 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-grpc-tls" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676354 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a9d6b0-dd41-4d54-bf35-110826270d66" volumeName="kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 02 22:53:26.676356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676366 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe3683b5-cfe1-408e-be5b-5a81c36304c4" volumeName="kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-metrics-client-ca" seLinuxMountContext="" Nov 02 22:53:26.677577 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676379 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe3683b5-cfe1-408e-be5b-5a81c36304c4" volumeName="kubernetes.io/empty-dir/fe3683b5-cfe1-408e-be5b-5a81c36304c4-volume-directive-shadow" seLinuxMountContext="" Nov 02 22:53:26.677577 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676392 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a4f8b44-11be-4522-806a-87f1e54ba005" volumeName="kubernetes.io/secret/2a4f8b44-11be-4522-806a-87f1e54ba005-cert" seLinuxMountContext="" Nov 02 22:53:26.677577 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676405 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41883b9a-e520-4e4d-8753-4512e53ff141" volumeName="kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-tls" seLinuxMountContext="" Nov 02 22:53:26.677577 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676416 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d158803-71d8-447b-8e4d-e73287682505" volumeName="kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-grpc-tls" seLinuxMountContext="" Nov 02 22:53:26.677577 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676428 5672 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526" volumeName="kubernetes.io/projected/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-kube-api-access-jkstf" seLinuxMountContext="" Nov 02 22:53:26.677577 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676439 5672 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:53:26.677577 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.676447 5672 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:53:26.703447 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.703414 5672 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:53:26.704851 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.704828 5672 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:53:26.704851 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.704851 5672 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:53:26.705054 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.704871 5672 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:53:26.705054 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:26.704914 5672 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 22:53:26.730274 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.730241 5672 generic.go:356] "Generic (PLEG): container finished" podID="fb8b21576a61098f70fabb74f44c4beb" containerID="27c2992b77b9d0fd7a46d8e79f8cc5925194f302658f65915a10709da67c719b" exitCode=0 Nov 02 22:53:26.750603 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.750571 5672 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="1bf23784aae8c843e6136fe839823452fdfaaf69aa4b0a0ee81e621cd2750aae" exitCode=0 Nov 02 22:53:26.750603 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.750594 5672 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="53f12615f0cac7c3e4705170f6bb47573e7c14965891089b2ddf984f7778d3e7" exitCode=0 Nov 02 22:53:26.750603 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.750601 5672 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="97d0c2aec72bd37d6a2054de2c4d9d4349b0f591de7281791d2b8e0729e120f5" exitCode=0 Nov 02 22:53:26.750806 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.750609 5672 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="ec742a2d51d4b5375c6d8ea1087467538e985e7cfffab8dfeb7aca00aac5206f" exitCode=0 Nov 02 22:53:26.750806 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.750620 5672 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="5b3c1bfc04c33f4d4f65657de1661e07dbfd0b0a540c8e2d2969cc80c72bffdf" exitCode=0 Nov 02 22:53:26.750806 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.750628 5672 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4d29-9676-4fff-a0cb-a26179a61e50" containerID="707e3785a569b84beac2853f3524fa3fda6e6aeac551f68edd91bd22943715f5" exitCode=0 Nov 02 22:53:26.759610 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.759587 5672 generic.go:356] "Generic (PLEG): container finished" podID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerID="89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c" exitCode=0 Nov 02 22:53:26.761817 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.761794 5672 generic.go:356] "Generic (PLEG): container finished" podID="41883b9a-e520-4e4d-8753-4512e53ff141" containerID="bf2d19b016df5edaef14cc792e217293d5674965e9e78ee136624e87f8d3cc9f" exitCode=0 Nov 02 22:53:26.772012 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:26.771989 5672 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:53:26.785260 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.785243 5672 manager.go:324] Recovery completed Nov 02 22:53:26.805033 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:26.804977 5672 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 22:53:26.805898 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.805884 5672 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:53:26.807839 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.807822 5672 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:53:26.807904 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.807850 5672 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:53:26.807904 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.807861 5672 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:53:26.810642 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810627 5672 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:53:26.810642 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810639 5672 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:53:26.810726 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810643 5672 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:53:26.810726 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810655 5672 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:53:26.810792 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810764 5672 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 22:53:26.810792 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810774 5672 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 22:53:26.810792 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810789 5672 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 22:53:26.810902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810795 5672 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 22:53:26.810902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810800 5672 policy_none.go:50] "Start" Nov 02 22:53:26.810902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810809 5672 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:53:26.810902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810817 5672 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:53:26.811014 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810954 5672 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 22:53:26.811014 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810962 5672 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 22:53:26.811014 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.810968 5672 policy_none.go:44] "Start" Nov 02 22:53:26.811516 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.811502 5672 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:53:26.811579 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.811527 5672 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:53:26.811579 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.811542 5672 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:53:26.815552 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.815528 5672 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:53:26.818081 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.818042 5672 manager.go:341] "Starting Device Plugin manager" Nov 02 22:53:26.818081 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:26.818064 5672 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:53:26.818081 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.818073 5672 server.go:87] "Starting device plugin registration server" Nov 02 22:53:26.818359 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.818346 5672 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:53:26.818417 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.818358 5672 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:53:26.818486 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.818469 5672 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:53:26.818601 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.818567 5672 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:53:26.818658 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.818609 5672 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:53:26.818658 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.818616 5672 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:53:26.818768 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.818653 5672 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:53:26.818598049Z","UUID":"0fe9673d-aad2-428d-a52c-1e67c8aa4cdd","Handler":null,"Name":"","Endpoint":""} Nov 02 22:53:26.821092 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:26.820931 5672 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:53:26.821092 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:26.820978 5672 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-4.us-west-2.compute.internal\" not found" Nov 02 22:53:26.821315 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.821262 5672 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:53:26.821315 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.821297 5672 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:53:26.849438 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.849416 5672 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:53:26.849438 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.849418 5672 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:53:26.849929 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.849912 5672 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 22:53:26.850372 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.850356 5672 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:53:26.918799 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.918781 5672 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:53:26.919745 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.919731 5672 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:53:26.919807 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.919757 5672 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:53:26.919807 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.919767 5672 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-4.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:53:26.919878 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.919808 5672 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:26.942022 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:26.942000 5672 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:53:27.005999 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.005958 5672 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal"] Nov 02 22:53:27.006637 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.006597 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" event={"ID":"fb8b21576a61098f70fabb74f44c4beb","Type":"ContainerStarted","Data":"e2486101a425ee1ad164c33d3bf030ca07a60623928f5908f779740918c83ca6"} Nov 02 22:53:27.006637 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.006629 5672 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.006787 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.006644 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" event={"ID":"fb8b21576a61098f70fabb74f44c4beb","Type":"ContainerDied","Data":"27c2992b77b9d0fd7a46d8e79f8cc5925194f302658f65915a10709da67c719b"} Nov 02 22:53:27.006787 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.006666 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" event={"ID":"fb8b21576a61098f70fabb74f44c4beb","Type":"ContainerStarted","Data":"4d634617a9a6f7d958a1241de51b5b4acedff197bdbe5fd0acf84bab0000d5d5"} Nov 02 22:53:27.006787 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.006684 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" event={"ID":"4e49e94ef0b64e567c61a7cf70cf3f00","Type":"ContainerStarted","Data":"59f470574a254011528a975d3f9b0aedfdb7465dee9eb22c48795004e160fc8a"} Nov 02 22:53:27.006787 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.006702 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" event={"ID":"4e49e94ef0b64e567c61a7cf70cf3f00","Type":"ContainerStarted","Data":"5d3263b7b47ac08bcd8d88fe8109bf9403ecf5ca6adc100a11549c9a346ed4dc"} Nov 02 22:53:27.006985 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.006835 5672 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.038494 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.038278 5672 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.038494 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.038336 5672 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.066300 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.066240 5672 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:53:27.066300 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.066278 5672 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.066716 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.066702 5672 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:53:27.066835 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.066735 5672 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.151617 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.151590 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.151741 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.151629 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.151741 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.151658 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e49e94ef0b64e567c61a7cf70cf3f00-config\") pod \"kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"4e49e94ef0b64e567c61a7cf70cf3f00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.252513 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.252476 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.252513 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.252516 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e49e94ef0b64e567c61a7cf70cf3f00-config\") pod \"kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"4e49e94ef0b64e567c61a7cf70cf3f00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.252690 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.252532 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.252690 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.252565 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.252690 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.252579 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fb8b21576a61098f70fabb74f44c4beb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"fb8b21576a61098f70fabb74f44c4beb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.252690 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.252575 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e49e94ef0b64e567c61a7cf70cf3f00-config\") pod \"kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal\" (UID: \"4e49e94ef0b64e567c61a7cf70cf3f00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.654500 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.654467 5672 apiserver.go:52] "Watching apiserver" Nov 02 22:53:27.750534 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.750503 5672 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:53:27.751089 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.751068 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b8df694dc-d54td","openshift-multus/network-metrics-daemon-zkbf8","openshift-network-diagnostics/network-check-target-2wfzr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal","openshift-image-registry/node-ca-p6x2b","openshift-ingress-canary/ingress-canary-mn22n","openshift-dns/node-resolver-rbhdf","openshift-insights/insights-runtime-extractor-j2jn2","kube-system/global-pull-secret-syncer-9b6c4","openshift-monitoring/node-exporter-4vrnv","openshift-ovn-kubernetes/ovnkube-node-pwwbw","kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal","openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz","kube-system/konnectivity-agent-m2smd","openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn","openshift-multus/multus-bfsqm","openshift-cluster-node-tuning-operator/tuned-527vp","openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx","openshift-network-operator/iptables-alerter-xv5ps","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc","openshift-dns/dns-default-wp45h","openshift-multus/multus-additional-cni-plugins-cqbm6","openshift-console/console-f695d65dd-478hr","openshift-monitoring/prometheus-k8s-0"] Nov 02 22:53:27.771075 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.771051 5672 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.771248 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.771178 5672 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.775222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.775204 5672 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:53:27.801236 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801213 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:53:27.801364 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801213 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9p98l\"" Nov 02 22:53:27.801364 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801289 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:53:27.801364 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801353 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:53:27.801510 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801361 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:53:27.801510 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801396 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:53:27.801510 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801405 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:53:27.801510 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801411 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:53:27.801510 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801423 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:53:27.801510 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801451 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77brg\"" Nov 02 22:53:27.801510 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801462 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:53:27.801710 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801567 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2rjj4\"" Nov 02 22:53:27.801710 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.801582 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:53:27.802017 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802001 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:53:27.802129 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802118 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:53:27.802417 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802402 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qgnp\"" Nov 02 22:53:27.802562 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802541 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:53:27.802562 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802554 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sdz5j\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802550 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c9lzr\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802576 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802664 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802711 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802722 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802737 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802739 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802776 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802783 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-92nqq\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802768 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802797 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802808 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802906 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802921 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v5hvl\"" Nov 02 22:53:27.802950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.802938 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:53:27.806054 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.805999 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:53:27.806054 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.806008 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5kxql\"" Nov 02 22:53:27.806269 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.806072 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n2bfg\"" Nov 02 22:53:27.806269 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.806229 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:53:27.806479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.806275 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:53:27.806479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.806289 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:53:27.806479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.806434 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-crtkq\"" Nov 02 22:53:27.806637 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.806539 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:53:27.806670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.806640 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:53:27.806670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.806652 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rxqrv\"" Nov 02 22:53:27.807337 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.807318 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:53:27.807466 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.807450 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:53:27.807516 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.807460 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:53:27.807562 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.807548 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:53:27.807631 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.807614 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:53:27.807692 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.807677 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vbflc\"" Nov 02 22:53:27.807742 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.807693 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:53:27.807817 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.807804 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8zgv4\"" Nov 02 22:53:27.845569 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.845543 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:53:27.845624 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.845598 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 22:53:27.845703 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.845688 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 22:53:27.845751 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.845703 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mvxcq\"" Nov 02 22:53:27.845987 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.845972 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 22:53:27.846141 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.846123 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 22:53:27.846235 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.846211 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 22:53:27.846302 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.846237 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m8ggg\"" Nov 02 22:53:27.846302 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.846249 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 22:53:27.846395 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.846322 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 22:53:27.846395 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.846361 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:53:27.846499 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.846487 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 22:53:27.846571 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.846561 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 22:53:27.846879 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.846867 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b9iueu8j41tm9\"" Nov 02 22:53:27.855518 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855501 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-node-log\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.855577 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855529 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:27.855577 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855548 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe3683b5-cfe1-408e-be5b-5a81c36304c4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.855577 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855564 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2j4\" (UniqueName: \"kubernetes.io/projected/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-api-access-4f2j4\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.855704 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855580 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.855704 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855596 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cnibin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.855704 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855610 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-k8s-cni-cncf-io\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.855704 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.855626 5672 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:27.855704 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855632 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-wtmp\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.855704 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855692 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe3683b5-cfe1-408e-be5b-5a81c36304c4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.855927 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855714 5672 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:53:27.855927 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.855818 5672 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.855927 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.855847 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:28.355789733 +0000 UTC m=+1.763101559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:27.855927 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855887 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-metrics-client-ca\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.856107 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.855926 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75cd1116-7b34-46e7-9502-fa71e12bbd9c-config-volume\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:53:27.856107 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856035 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-sys\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.856107 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856066 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-netd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.856264 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856095 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.856300 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856244 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-modprobe-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.856336 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856306 5672 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:53:27.856336 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856311 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.856408 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856349 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-device-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.856408 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.856364 5672 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-4.us-west-2.compute.internal" Nov 02 22:53:27.856408 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856373 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-bin\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.856534 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856402 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-system-cni-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.856534 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856456 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:27.856534 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856478 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-conf\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.856664 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856620 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-systemd\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.856729 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856690 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-tuned\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.856806 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856788 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-tuned\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.856854 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856814 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:27.856854 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856835 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.856929 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856876 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mrv\" (UniqueName: \"kubernetes.io/projected/b8238d30-1c60-4c49-9171-881a3f46f399-kube-api-access-q7mrv\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:53:27.856929 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856896 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-config\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.856929 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856920 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-socket-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.857059 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.856965 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrjz\" (UniqueName: \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:27.857099 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857058 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.857143 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857106 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.857210 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857135 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-kubelet\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.857210 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857191 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-multus-certs\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.857305 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857212 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.857403 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857385 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-conf-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.857459 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857420 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-textfile\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.857459 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857445 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysconfig\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.857550 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857469 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-socket-dir-parent\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.857550 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857511 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:53:27.857624 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857557 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a4f8b44-11be-4522-806a-87f1e54ba005-cert\") pod \"ingress-canary-mn22n\" (UID: \"2a4f8b44-11be-4522-806a-87f1e54ba005\") " pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:53:27.857624 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857584 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-textfile\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.857624 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857590 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-accelerators-collector-config\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.857729 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857626 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtxmf\" (UniqueName: \"kubernetes.io/projected/75cd1116-7b34-46e7-9502-fa71e12bbd9c-kube-api-access-dtxmf\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:53:27.857729 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857648 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-lib-modules\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.857729 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857663 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.857729 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857677 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-tls\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.857729 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857700 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjjh\" (UniqueName: \"kubernetes.io/projected/7d17af01-21f0-48b5-b6b3-834a2e2c7852-kube-api-access-pqjjh\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:27.857939 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857855 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a4f8b44-11be-4522-806a-87f1e54ba005-cert\") pod \"ingress-canary-mn22n\" (UID: \"2a4f8b44-11be-4522-806a-87f1e54ba005\") " pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:53:27.857983 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857937 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-accelerators-collector-config\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.857983 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857944 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtz2w\" (UniqueName: \"kubernetes.io/projected/ae18bf8e-ca0b-41e8-90ca-9e830160012a-kube-api-access-jtz2w\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:53:27.858052 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857979 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.858052 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.857937 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad49b548-e79b-43aa-9479-57c1e6d06fdd-original-pull-secret\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:53:27.858052 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.858033 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-systemd-units\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.858169 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.858073 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:27.858230 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.858196 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.858262 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.858232 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-tls\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.858262 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.858248 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcmgd\" (UniqueName: \"kubernetes.io/projected/c8ae4d29-9676-4fff-a0cb-a26179a61e50-kube-api-access-hcmgd\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.858323 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.858214 5672 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 02 22:53:27.858323 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.858280 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-etc-selinux\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.858774 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.858748 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.858839 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.858459 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovn-node-metrics-cert\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.858839 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.858808 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:28.358760468 +0000 UTC m=+1.766072289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"service-ca" not registered Nov 02 22:53:27.858942 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.858885 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8238d30-1c60-4c49-9171-881a3f46f399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:53:27.858942 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.858932 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-var-lib-kubelet\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.859047 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.858966 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-host\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.859047 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859009 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.859133 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859044 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:27.859133 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859037 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovn-node-metrics-cert\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.859239 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.859190 5672 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:27.859286 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859252 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8238d30-1c60-4c49-9171-881a3f46f399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:53:27.859286 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859266 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8238d30-1c60-4c49-9171-881a3f46f399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:53:27.859381 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.859285 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:28.359265312 +0000 UTC m=+1.766577143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:27.859381 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859360 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-tls\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.859494 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859394 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.859544 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859521 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8238d30-1c60-4c49-9171-881a3f46f399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:53:27.859544 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859534 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.859739 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859719 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-tls\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.859801 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859765 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-binary-copy\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.859857 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859818 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d17af01-21f0-48b5-b6b3-834a2e2c7852-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:27.859906 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859866 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtsgl\" (UniqueName: \"kubernetes.io/projected/526ff9d4-f30b-4414-849b-c3e747329dd9-kube-api-access-xtsgl\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.859955 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859904 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-ovn\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.860003 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.859948 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kbgs\" (UniqueName: \"kubernetes.io/projected/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-kube-api-access-9kbgs\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.860060 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860008 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8238d30-1c60-4c49-9171-881a3f46f399-crio-socket\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:53:27.860060 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860051 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.860175 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860079 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.860175 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860087 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.860283 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860134 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.860283 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860230 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6w4k\" (UniqueName: \"kubernetes.io/projected/2a4f8b44-11be-4522-806a-87f1e54ba005-kube-api-access-r6w4k\") pod \"ingress-canary-mn22n\" (UID: \"2a4f8b44-11be-4522-806a-87f1e54ba005\") " pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:53:27.860662 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860635 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.860741 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860689 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.860741 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860726 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.860849 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860758 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-multus\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.860849 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860802 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.860849 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860841 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.860993 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860879 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-systemd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.860993 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860920 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:27.860993 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860955 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.861137 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860998 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.861137 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.860993 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-konnectivity-ca\") pod \"konnectivity-agent-m2smd\" (UID: \"52e9c219-9e54-43c4-9dae-7d60c8e33dfe\") " pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:53:27.861137 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861049 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d158803-71d8-447b-8e4d-e73287682505-metrics-client-ca\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.861137 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861084 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-tmp\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.861348 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861136 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.861348 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861198 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/526ff9d4-f30b-4414-849b-c3e747329dd9-tmp\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.861348 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861220 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.861348 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861256 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:27.861348 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861282 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.861348 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.861302 5672 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:27.861348 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861313 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbf65\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-kube-api-access-qbf65\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.861670 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.861357 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:28.361335728 +0000 UTC m=+1.768647553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:27.861670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861380 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-registration-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.861670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861418 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-log-socket\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.861670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861455 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-netns\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.861670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861487 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:53:27.861670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861522 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.861670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861557 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae18bf8e-ca0b-41e8-90ca-9e830160012a-hosts-file\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:53:27.861670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861594 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.861670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861622 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.862087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861678 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-daemon-config\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.862087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861713 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlv6j\" (UniqueName: \"kubernetes.io/projected/889d0ebb-164e-4c18-973a-509f4dcfa1bc-kube-api-access-tlv6j\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.862087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861747 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8238d30-1c60-4c49-9171-881a3f46f399-data-volume\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:53:27.862087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861749 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.862087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.861995 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-konnectivity-ca\") pod \"konnectivity-agent-m2smd\" (UID: \"52e9c219-9e54-43c4-9dae-7d60c8e33dfe\") " pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:53:27.862352 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862253 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d17af01-21f0-48b5-b6b3-834a2e2c7852-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:27.862406 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862358 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxx5\" (UniqueName: \"kubernetes.io/projected/4d158803-71d8-447b-8e4d-e73287682505-kube-api-access-nqxx5\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.862406 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862396 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.862508 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862409 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.862508 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862430 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-kubelet\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.862508 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862456 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-var-lib-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.862508 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862485 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-etc-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.862688 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862516 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:27.862688 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862565 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrnjr\" (UniqueName: \"kubernetes.io/projected/0798910d-3562-4dfe-b069-587510b73467-kube-api-access-qrnjr\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:53:27.862688 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.862579 5672 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 02 22:53:27.862688 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862608 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkstf\" (UniqueName: \"kubernetes.io/projected/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-kube-api-access-jkstf\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:53:27.862688 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.862648 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:28.362628462 +0000 UTC m=+1.769940284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-config" not registered Nov 02 22:53:27.862688 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862666 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae18bf8e-ca0b-41e8-90ca-9e830160012a-tmp-dir\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:53:27.862966 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862699 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-config\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.862966 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862731 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-grpc-tls\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.862966 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862770 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.862966 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862800 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-slash\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.862966 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.862829 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aeea33da-f0ef-4912-8ebc-29b48d336448-tmp\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:27.863273 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.863001 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae18bf8e-ca0b-41e8-90ca-9e830160012a-tmp-dir\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:53:27.863273 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.863042 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-daemon-config\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.863273 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.863231 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-config\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.863743 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.863526 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-grpc-tls\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.863743 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.863555 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8238d30-1c60-4c49-9171-881a3f46f399-data-volume\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:53:27.863743 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.863723 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.863921 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.863796 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqkgp\" (UniqueName: \"kubernetes.io/projected/41883b9a-e520-4e4d-8753-4512e53ff141-kube-api-access-pqkgp\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.863921 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.863867 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:53:27.863921 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.863881 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aeea33da-f0ef-4912-8ebc-29b48d336448-tmp\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:27.864053 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864019 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-script-lib\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.864109 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864100 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-iptables-alerter-script\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:53:27.864170 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864128 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-host\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:53:27.864220 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864169 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-serviceca\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:53:27.864330 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864291 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkvpl\" (UniqueName: \"kubernetes.io/projected/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-kube-api-access-gkvpl\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:53:27.864665 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864644 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.864737 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864681 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cni-binary-copy\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.864737 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864697 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-iptables-alerter-script\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:53:27.864737 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864711 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-hostroot\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.864737 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864723 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-ovnkube-script-lib\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.864930 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864696 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0798910d-3562-4dfe-b069-587510b73467-metrics-certs\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:53:27.864930 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864784 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pnv7\" (UniqueName: \"kubernetes.io/projected/e26c39e8-2ffc-4179-8498-76167f635e9a-kube-api-access-4pnv7\") pod \"monitoring-plugin-77f4f76594-zhjrz\" (UID: \"e26c39e8-2ffc-4179-8498-76167f635e9a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:27.864930 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864789 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-config-out\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.864930 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864882 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.864930 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864910 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-bin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.865134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864970 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.865134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.864980 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cni-binary-copy\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.865134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865000 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-env-overrides\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.865134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865032 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.865134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865059 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-root\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.865134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865091 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-host-slash\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:53:27.865134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865111 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-env-overrides\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.865134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865133 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cnibin\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.865375 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865190 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-kubernetes\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.865375 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865209 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-serviceca\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:53:27.865375 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865230 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-run\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.865375 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865260 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.865375 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865289 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-netns\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.865375 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865307 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-sys\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.865375 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865340 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.865568 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865370 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e26c39e8-2ffc-4179-8498-76167f635e9a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zhjrz\" (UID: \"e26c39e8-2ffc-4179-8498-76167f635e9a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:27.865568 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865409 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-system-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.865568 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865439 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-dbus\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:53:27.865568 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865469 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-agent-certs\") pod \"konnectivity-agent-m2smd\" (UID: \"52e9c219-9e54-43c4-9dae-7d60c8e33dfe\") " pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:53:27.865568 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865502 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75cd1116-7b34-46e7-9502-fa71e12bbd9c-metrics-tls\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:53:27.865568 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865524 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.865568 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865527 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75cd1116-7b34-46e7-9502-fa71e12bbd9c-tmp-dir\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:53:27.865568 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865556 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-sys-fs\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.865778 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865572 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-etc-kubernetes\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.865778 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865588 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-kubelet-config\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:53:27.865778 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865600 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75cd1116-7b34-46e7-9502-fa71e12bbd9c-tmp-dir\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:53:27.865778 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865611 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d158803-71d8-447b-8e4d-e73287682505-tmp\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.865778 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865642 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.865778 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865651 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d158803-71d8-447b-8e4d-e73287682505-tmp\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.865778 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865671 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97g84\" (UniqueName: \"kubernetes.io/projected/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kube-api-access-97g84\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.865957 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865800 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/75cd1116-7b34-46e7-9502-fa71e12bbd9c-metrics-tls\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:53:27.865957 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.865838 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/52e9c219-9e54-43c4-9dae-7d60c8e33dfe-agent-certs\") pod \"konnectivity-agent-m2smd\" (UID: \"52e9c219-9e54-43c4-9dae-7d60c8e33dfe\") " pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:53:27.900498 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.900471 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 22:53:27.900498 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.900493 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 22:53:27.900651 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.900506 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 22:53:27.900923 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.900870 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rnj8n\"" Nov 02 22:53:27.900923 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.900911 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:53:27.901038 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.900937 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 22:53:27.901252 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901233 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 22:53:27.901348 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901287 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 22:53:27.901348 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901286 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:53:27.901348 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901306 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 22:53:27.901348 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901321 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-75wn4\"" Nov 02 22:53:27.901495 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901474 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 22:53:27.901657 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901640 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 22:53:27.901712 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901695 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 22:53:27.901769 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901723 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 22:53:27.901769 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901754 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 22:53:27.901864 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901778 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 22:53:27.901864 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901808 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 22:53:27.901864 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901828 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 22:53:27.901864 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901756 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 22:53:27.902034 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901884 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.902034 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901892 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:53:27.902034 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901899 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wkpfp\"" Nov 02 22:53:27.902034 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901980 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6gjv\"" Nov 02 22:53:27.902034 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.901988 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-web-config\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.902273 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.902121 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 22:53:27.902331 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.902314 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r7xnr\"" Nov 02 22:53:27.902387 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.902374 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 22:53:27.902426 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.902420 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 22:53:27.902636 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.902624 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-tedh3m60u5r0\"" Nov 02 22:53:27.902820 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.902802 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.902926 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.902910 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 22:53:27.903186 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.903173 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 22:53:27.903976 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.903958 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.906213 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.906160 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.906213 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.906173 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3683b5-cfe1-408e-be5b-5a81c36304c4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.906213 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.906195 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41883b9a-e520-4e4d-8753-4512e53ff141-metrics-client-ca\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.906393 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.906239 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e26c39e8-2ffc-4179-8498-76167f635e9a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zhjrz\" (UID: \"e26c39e8-2ffc-4179-8498-76167f635e9a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:27.906393 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.906301 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.906393 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.906331 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/75cd1116-7b34-46e7-9502-fa71e12bbd9c-config-volume\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:53:27.906393 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.906341 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.906393 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.906387 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.906533 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.906445 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.906533 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.906526 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.907296 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.907272 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-config\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.907377 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.907308 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.907466 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.907450 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.907562 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.907541 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.907826 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.907811 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.908477 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.908463 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.908564 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.908551 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.909298 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.909282 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2j4\" (UniqueName: \"kubernetes.io/projected/fe3683b5-cfe1-408e-be5b-5a81c36304c4-kube-api-access-4f2j4\") pod \"kube-state-metrics-5dc8d7d89b-qvnbn\" (UID: \"fe3683b5-cfe1-408e-be5b-5a81c36304c4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qvnbn" Nov 02 22:53:27.910270 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.910255 5672 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 02 22:53:27.910313 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.910276 5672 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 02 22:53:27.910313 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.910285 5672 projected.go:196] Error preparing data for projected volume kube-api-access-qlrjz for pod openshift-console/console-f695d65dd-478hr: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:27.910405 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:27.910356 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:28.410344387 +0000 UTC m=+1.817656200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qlrjz" (UniqueName: "kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:27.910496 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.910481 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.910552 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.910509 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d17af01-21f0-48b5-b6b3-834a2e2c7852-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:27.910608 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.910587 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7mrv\" (UniqueName: \"kubernetes.io/projected/b8238d30-1c60-4c49-9171-881a3f46f399-kube-api-access-q7mrv\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:53:27.910739 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.910721 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtz2w\" (UniqueName: \"kubernetes.io/projected/ae18bf8e-ca0b-41e8-90ca-9e830160012a-kube-api-access-jtz2w\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:53:27.911226 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.911207 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d158803-71d8-447b-8e4d-e73287682505-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.911310 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.911259 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcmgd\" (UniqueName: \"kubernetes.io/projected/c8ae4d29-9676-4fff-a0cb-a26179a61e50-kube-api-access-hcmgd\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.911614 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.911598 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjjh\" (UniqueName: \"kubernetes.io/projected/7d17af01-21f0-48b5-b6b3-834a2e2c7852-kube-api-access-pqjjh\") pod \"openshift-state-metrics-7bd7489957-lh4jx\" (UID: \"7d17af01-21f0-48b5-b6b3-834a2e2c7852\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lh4jx" Nov 02 22:53:27.912239 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.912218 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtxmf\" (UniqueName: \"kubernetes.io/projected/75cd1116-7b34-46e7-9502-fa71e12bbd9c-kube-api-access-dtxmf\") pod \"dns-default-wp45h\" (UID: \"75cd1116-7b34-46e7-9502-fa71e12bbd9c\") " pod="openshift-dns/dns-default-wp45h" Nov 02 22:53:27.912343 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.912319 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d158803-71d8-447b-8e4d-e73287682505-metrics-client-ca\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.912493 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.912476 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.912843 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.912826 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6w4k\" (UniqueName: \"kubernetes.io/projected/2a4f8b44-11be-4522-806a-87f1e54ba005-kube-api-access-r6w4k\") pod \"ingress-canary-mn22n\" (UID: \"2a4f8b44-11be-4522-806a-87f1e54ba005\") " pod="openshift-ingress-canary/ingress-canary-mn22n" Nov 02 22:53:27.912958 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.912942 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtsgl\" (UniqueName: \"kubernetes.io/projected/526ff9d4-f30b-4414-849b-c3e747329dd9-kube-api-access-xtsgl\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.913261 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.913242 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pcfv\" (UniqueName: \"kubernetes.io/projected/b37eb397-e28e-49c2-a711-a4efd3466679-kube-api-access-7pcfv\") pod \"network-check-target-2wfzr\" (UID: \"b37eb397-e28e-49c2-a711-a4efd3466679\") " pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:53:27.913296 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.913278 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kbgs\" (UniqueName: \"kubernetes.io/projected/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-kube-api-access-9kbgs\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.913515 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.913494 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbf65\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-kube-api-access-qbf65\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.913761 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.913747 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkstf\" (UniqueName: \"kubernetes.io/projected/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-kube-api-access-jkstf\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:53:27.914426 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.914405 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxx5\" (UniqueName: \"kubernetes.io/projected/4d158803-71d8-447b-8e4d-e73287682505-kube-api-access-nqxx5\") pod \"thanos-querier-7b8df694dc-d54td\" (UID: \"4d158803-71d8-447b-8e4d-e73287682505\") " pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:27.915589 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.915570 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:27.915739 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.915721 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrnjr\" (UniqueName: \"kubernetes.io/projected/0798910d-3562-4dfe-b069-587510b73467-kube-api-access-qrnjr\") pod \"network-metrics-daemon-zkbf8\" (UID: \"0798910d-3562-4dfe-b069-587510b73467\") " pod="openshift-multus/network-metrics-daemon-zkbf8" Nov 02 22:53:27.916026 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.916006 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqkgp\" (UniqueName: \"kubernetes.io/projected/41883b9a-e520-4e4d-8753-4512e53ff141-kube-api-access-pqkgp\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.916283 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.916268 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pnv7\" (UniqueName: \"kubernetes.io/projected/e26c39e8-2ffc-4179-8498-76167f635e9a-kube-api-access-4pnv7\") pod \"monitoring-plugin-77f4f76594-zhjrz\" (UID: \"e26c39e8-2ffc-4179-8498-76167f635e9a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:27.916828 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.916812 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlv6j\" (UniqueName: \"kubernetes.io/projected/889d0ebb-164e-4c18-973a-509f4dcfa1bc-kube-api-access-tlv6j\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.916895 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.916880 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97g84\" (UniqueName: \"kubernetes.io/projected/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kube-api-access-97g84\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.916927 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.916880 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkvpl\" (UniqueName: \"kubernetes.io/projected/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-kube-api-access-gkvpl\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:53:27.966765 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966739 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-lib-modules\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.966868 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966773 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-systemd-units\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.966868 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966808 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-etc-selinux\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.966868 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966858 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-systemd-units\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.966986 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966871 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-etc-selinux\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.966986 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966897 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-var-lib-kubelet\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.966986 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966924 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-host\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.966986 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966921 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-lib-modules\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.966986 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966953 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.966986 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966984 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-host\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966999 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.966990 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-var-lib-kubelet\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967028 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-ovn\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967004 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-ovn\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967058 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8238d30-1c60-4c49-9171-881a3f46f399-crio-socket\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967088 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967106 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967109 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8238d30-1c60-4c49-9171-881a3f46f399-crio-socket\") pod \"insights-runtime-extractor-j2jn2\" (UID: \"b8238d30-1c60-4c49-9171-881a3f46f399\") " pod="openshift-insights/insights-runtime-extractor-j2jn2" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967124 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-multus\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967141 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-systemd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967177 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967192 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967193 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-multus\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967216 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967222 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967232 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-run-systemd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967244 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-registration-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967269 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-ovn-kubernetes\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967271 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-log-socket\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967295 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-registration-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967303 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-log-socket\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967305 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-netns\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967323 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-netns\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967342 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae18bf8e-ca0b-41e8-90ca-9e830160012a-hosts-file\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967367 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967393 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae18bf8e-ca0b-41e8-90ca-9e830160012a-hosts-file\") pod \"node-resolver-rbhdf\" (UID: \"ae18bf8e-ca0b-41e8-90ca-9e830160012a\") " pod="openshift-dns/node-resolver-rbhdf" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967402 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967401 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-kubelet\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967436 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-var-lib-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967451 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-etc-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967434 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-kubelet\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967476 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-var-lib-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967487 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967499 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-etc-openvswitch\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967504 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-slash\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967537 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-host\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967553 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-hostroot\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967572 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78-host\") pod \"node-ca-p6x2b\" (UID: \"a2a82c3d-2c8e-4250-b6fd-09e55c4d7b78\") " pod="openshift-image-registry/node-ca-p6x2b" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967576 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-bin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967578 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-slash\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967593 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967595 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-hostroot\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967609 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-root\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967611 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967624 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-cni-bin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967626 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-host-slash\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967638 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967642 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-root\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967648 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cnibin\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967665 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526-host-slash\") pod \"iptables-alerter-xv5ps\" (UID: \"546959a7-c5ae-4ea7-a9b7-c6ed4c0d0526\") " pod="openshift-network-operator/iptables-alerter-xv5ps" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967670 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-cnibin\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967680 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-kubernetes\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967705 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-run\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967722 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-netns\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967740 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-sys\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967756 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-system-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967748 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-kubernetes\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967767 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-run\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967785 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-dbus\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967799 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-run-netns\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967810 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-sys-fs\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967817 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-system-cni-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967787 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-sys\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967832 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-dbus\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967830 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-etc-kubernetes\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967849 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-etc-kubernetes\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967870 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-kubelet-config\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967846 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-sys-fs\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.967875 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967902 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-node-log\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967917 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad49b548-e79b-43aa-9479-57c1e6d06fdd-kubelet-config\") pod \"global-pull-secret-syncer-9b6c4\" (UID: \"ad49b548-e79b-43aa-9479-57c1e6d06fdd\") " pod="kube-system/global-pull-secret-syncer-9b6c4" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967928 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cnibin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967942 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-k8s-cni-cncf-io\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967973 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-node-log\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967988 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-wtmp\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968004 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-sys\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968007 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-k8s-cni-cncf-io\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.967977 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-cnibin\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968018 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-netd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968033 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-modprobe-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968048 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-device-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968055 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/41883b9a-e520-4e4d-8753-4512e53ff141-node-exporter-wtmp\") pod \"node-exporter-4vrnv\" (UID: \"41883b9a-e520-4e4d-8753-4512e53ff141\") " pod="openshift-monitoring/node-exporter-4vrnv" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968062 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-bin\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968073 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-sys\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968087 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-netd\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968105 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-device-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968104 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-system-cni-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968126 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/584a133a-3ca7-4a1b-88fc-83d98e3abc3d-host-cni-bin\") pod \"ovnkube-node-pwwbw\" (UID: \"584a133a-3ca7-4a1b-88fc-83d98e3abc3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968131 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-conf\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968133 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8ae4d29-9676-4fff-a0cb-a26179a61e50-system-cni-dir\") pod \"multus-additional-cni-plugins-cqbm6\" (UID: \"c8ae4d29-9676-4fff-a0cb-a26179a61e50\") " pod="openshift-multus/multus-additional-cni-plugins-cqbm6" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968166 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-systemd\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968171 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-modprobe-d\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968186 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-socket-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968209 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-kubelet\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968214 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-systemd\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968229 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-multus-certs\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968252 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-conf-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968258 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-var-lib-kubelet\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968266 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce280d50-8be5-4223-91ce-3ea1da4e7ff6-socket-dir\") pod \"aws-ebs-csi-driver-node-xb2lc\" (UID: \"ce280d50-8be5-4223-91ce-3ea1da4e7ff6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xb2lc" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968277 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysconfig\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968263 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysctl-conf\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968289 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-host-run-multus-certs\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968302 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-socket-dir-parent\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968320 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/526ff9d4-f30b-4414-849b-c3e747329dd9-etc-sysconfig\") pod \"tuned-527vp\" (UID: \"526ff9d4-f30b-4414-849b-c3e747329dd9\") " pod="openshift-cluster-node-tuning-operator/tuned-527vp" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968310 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-conf-dir\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:27.969441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:27.968347 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/889d0ebb-164e-4c18-973a-509f4dcfa1bc-multus-socket-dir-parent\") pod \"multus-bfsqm\" (UID: \"889d0ebb-164e-4c18-973a-509f4dcfa1bc\") " pod="openshift-multus/multus-bfsqm" Nov 02 22:53:28.166084 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:28.166013 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zhjrz" Nov 02 22:53:28.371485 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:28.371454 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:28.371637 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:28.371491 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:28.371637 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:28.371519 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:28.371637 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:28.371545 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:28.371637 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.371591 5672 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 02 22:53:28.371637 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.371609 5672 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:28.371891 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.371641 5672 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:28.371891 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:28.371599 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:28.371891 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.371660 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:29.371642424 +0000 UTC m=+2.778954249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"service-ca" not registered Nov 02 22:53:28.371891 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.371641 5672 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:28.371891 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.371692 5672 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 02 22:53:28.371891 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.371736 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:29.371721161 +0000 UTC m=+2.779032985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:28.371891 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.371758 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:29.371752244 +0000 UTC m=+2.779064052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:28.371891 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.371770 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:29.371765023 +0000 UTC m=+2.779076831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:28.371891 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.371786 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:29.371778963 +0000 UTC m=+2.779090771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-config" not registered Nov 02 22:53:28.441525 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:28.441461 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:28.472874 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:28.472834 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrjz\" (UniqueName: \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:28.473021 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.472992 5672 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 02 22:53:28.473021 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.473009 5672 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 02 22:53:28.473021 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.473019 5672 projected.go:196] Error preparing data for projected volume kube-api-access-qlrjz for pod openshift-console/console-f695d65dd-478hr: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:28.473113 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:28.473067 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:29.473052431 +0000 UTC m=+2.880364239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlrjz" (UniqueName: "kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:28.907986 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:28.907960 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b8df694dc-d54td" Nov 02 22:53:29.381436 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:29.381403 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:29.381588 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:29.381455 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:29.381588 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:29.381497 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:29.381588 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:29.381518 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:29.381588 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.381520 5672 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 02 22:53:29.381588 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:29.381545 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:29.381588 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.381581 5672 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:29.381830 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.381603 5672 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 02 22:53:29.381830 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.381582 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:31.381568021 +0000 UTC m=+4.788879851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-config" not registered Nov 02 22:53:29.381830 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.381647 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:31.381633431 +0000 UTC m=+4.788945239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:29.381830 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.381648 5672 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:29.381830 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.381651 5672 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:29.381830 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.381662 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:31.38165292 +0000 UTC m=+4.788964728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"service-ca" not registered Nov 02 22:53:29.381830 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.381684 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:31.381672916 +0000 UTC m=+4.788984743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:29.381830 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.381711 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:31.38169072 +0000 UTC m=+4.789002528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:29.419015 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:29.418989 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wp45h" Nov 02 22:53:29.482677 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:29.482656 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrjz\" (UniqueName: \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:29.482809 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.482774 5672 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 02 22:53:29.482809 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.482786 5672 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 02 22:53:29.482809 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.482796 5672 projected.go:196] Error preparing data for projected volume kube-api-access-qlrjz for pod openshift-console/console-f695d65dd-478hr: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:29.482901 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:29.482832 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:31.482823067 +0000 UTC m=+4.890134876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlrjz" (UniqueName: "kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:31.400931 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:31.400897 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:31.400965 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:31.400998 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.401013 5672 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:31.401037 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.401077 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:35.401060536 +0000 UTC m=+8.808372361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.401103 5672 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.401128 5672 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.401133 5672 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:31.401106 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.401139 5672 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.401192 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:35.401173978 +0000 UTC m=+8.808485801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"service-ca" not registered Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.401247 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:35.401230402 +0000 UTC m=+8.808542224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.401266 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:35.401257417 +0000 UTC m=+8.808569225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:31.401470 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.401284 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:35.401273881 +0000 UTC m=+8.808585690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-config" not registered Nov 02 22:53:31.501751 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:31.501718 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrjz\" (UniqueName: \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:31.501891 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.501838 5672 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 02 22:53:31.501891 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.501854 5672 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 02 22:53:31.501891 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.501864 5672 projected.go:196] Error preparing data for projected volume kube-api-access-qlrjz for pod openshift-console/console-f695d65dd-478hr: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:31.501991 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:31.501912 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:35.501895856 +0000 UTC m=+8.909207667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlrjz" (UniqueName: "kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:31.893264 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:31.893238 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:53:33.830659 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:33.830627 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:33.831005 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:33.830815 5672 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:53:35.440081 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:35.440046 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:35.440081 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:35.440082 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:35.440108 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:35.440144 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.440196 5672 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.440226 5672 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.440252 5672 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.440257 5672 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.440272 5672 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.440285 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:43.44026142 +0000 UTC m=+16.847573243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"service-ca" not registered Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.440306 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:43.440296889 +0000 UTC m=+16.847608698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:35.440210 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.440330 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:43.440314561 +0000 UTC m=+16.847626373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.440348 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:43.440342047 +0000 UTC m=+16.847653855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:35.440674 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.440366 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:43.440359224 +0000 UTC m=+16.847671032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-config" not registered Nov 02 22:53:35.541395 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:35.541363 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrjz\" (UniqueName: \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:35.541528 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.541495 5672 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 02 22:53:35.541528 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.541510 5672 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 02 22:53:35.541528 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.541520 5672 projected.go:196] Error preparing data for projected volume kube-api-access-qlrjz for pod openshift-console/console-f695d65dd-478hr: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:35.541641 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:35.541559 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:43.541547066 +0000 UTC m=+16.948858891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlrjz" (UniqueName: "kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:36.005393 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:36.005359 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:53:43.510804 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:43.510760 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:43.510804 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:43.510809 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:43.510834 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:43.510865 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.510882 5672 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:43.510898 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.510905 5672 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.510961 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:59.510939685 +0000 UTC m=+32.918251514 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"service-ca" not registered Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.510983 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:59.510972112 +0000 UTC m=+32.918283920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-oauth-config" not registered Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.510986 5672 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.510992 5672 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.510984 5672 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.511038 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:59.511021987 +0000 UTC m=+32.918333817 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-serving-cert" not registered Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.511085 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:59.511072508 +0000 UTC m=+32.918384317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"console-config" not registered Nov 02 22:53:43.511315 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.511097 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:59.511090791 +0000 UTC m=+32.918402599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : object "openshift-console"/"oauth-serving-cert" not registered Nov 02 22:53:43.611667 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:43.611641 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrjz\" (UniqueName: \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\") pod \"console-f695d65dd-478hr\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:43.611795 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.611770 5672 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Nov 02 22:53:43.611795 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.611785 5672 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Nov 02 22:53:43.611795 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.611795 5672 projected.go:196] Error preparing data for projected volume kube-api-access-qlrjz for pod openshift-console/console-f695d65dd-478hr: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:43.611892 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:43.611839 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz podName:aeea33da-f0ef-4912-8ebc-29b48d336448 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:59.611829192 +0000 UTC m=+33.019141001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlrjz" (UniqueName: "kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz") pod "console-f695d65dd-478hr" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Nov 02 22:53:44.418109 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:44.418077 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m2smd" Nov 02 22:53:48.596616 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:48.596585 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pwwbw" Nov 02 22:53:52.779238 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:52.779185 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f695d65dd-478hr" podUID="aeea33da-f0ef-4912-8ebc-29b48d336448" containerName="console" containerID="cri-o://64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb" gracePeriod=15 Nov 02 22:53:53.037224 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.037171 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:53.205602 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.205573 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\") pod \"aeea33da-f0ef-4912-8ebc-29b48d336448\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " Nov 02 22:53:53.205757 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.205622 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\") pod \"aeea33da-f0ef-4912-8ebc-29b48d336448\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " Nov 02 22:53:53.205757 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.205658 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\") pod \"aeea33da-f0ef-4912-8ebc-29b48d336448\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " Nov 02 22:53:53.205757 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.205683 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\" (UniqueName: \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\") pod \"aeea33da-f0ef-4912-8ebc-29b48d336448\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " Nov 02 22:53:53.205757 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.205705 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeea33da-f0ef-4912-8ebc-29b48d336448-tmp\" (UniqueName: \"kubernetes.io/empty-dir/aeea33da-f0ef-4912-8ebc-29b48d336448-tmp\") pod \"aeea33da-f0ef-4912-8ebc-29b48d336448\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " Nov 02 22:53:53.205935 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.205817 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\") pod \"aeea33da-f0ef-4912-8ebc-29b48d336448\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " Nov 02 22:53:53.205935 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.205867 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\") pod \"aeea33da-f0ef-4912-8ebc-29b48d336448\" (UID: \"aeea33da-f0ef-4912-8ebc-29b48d336448\") " Nov 02 22:53:53.206230 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.206131 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeea33da-f0ef-4912-8ebc-29b48d336448-tmp" pod "aeea33da-f0ef-4912-8ebc-29b48d336448" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:53:53.206230 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.206203 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert" pod "aeea33da-f0ef-4912-8ebc-29b48d336448" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:53.206397 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.206218 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca" pod "aeea33da-f0ef-4912-8ebc-29b48d336448" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:53.206438 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.206410 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config" pod "aeea33da-f0ef-4912-8ebc-29b48d336448" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:53.208203 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.208168 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert" pod "aeea33da-f0ef-4912-8ebc-29b48d336448" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:53:53.208358 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.208334 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz" pod "aeea33da-f0ef-4912-8ebc-29b48d336448" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448"). InnerVolumeSpecName "kube-api-access-qlrjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:53.208400 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.208342 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config" pod "aeea33da-f0ef-4912-8ebc-29b48d336448" (UID: "aeea33da-f0ef-4912-8ebc-29b48d336448"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:53:53.307392 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.307331 5672 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-oauth-serving-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:53.307392 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.307354 5672 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:53.307392 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.307363 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlrjz\" (UniqueName: \"kubernetes.io/projected/aeea33da-f0ef-4912-8ebc-29b48d336448-kube-api-access-qlrjz\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:53.307392 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.307374 5672 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aeea33da-f0ef-4912-8ebc-29b48d336448-tmp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:53.307392 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.307383 5672 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-serving-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:53.307392 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.307393 5672 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aeea33da-f0ef-4912-8ebc-29b48d336448-console-config\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:53.307659 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.307403 5672 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aeea33da-f0ef-4912-8ebc-29b48d336448-console-oauth-config\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:53.862767 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.862737 5672 generic.go:356] "Generic (PLEG): container finished" podID="aeea33da-f0ef-4912-8ebc-29b48d336448" containerID="64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb" exitCode=0 Nov 02 22:53:53.863237 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.862783 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f695d65dd-478hr" Nov 02 22:53:53.863237 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.862797 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f695d65dd-478hr" event={"ID":"aeea33da-f0ef-4912-8ebc-29b48d336448","Type":"ContainerDied","Data":"64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb"} Nov 02 22:53:53.863237 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.862847 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f695d65dd-478hr" event={"ID":"aeea33da-f0ef-4912-8ebc-29b48d336448","Type":"ContainerDied","Data":"9a4df370ea05876a66ed6da79c4b1f7e85906f89b6ffd07d4ba0b8ed637e490e"} Nov 02 22:53:53.863237 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.862876 5672 scope.go:122] "RemoveContainer" containerID="64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb" Nov 02 22:53:53.870636 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.870616 5672 scope.go:122] "RemoveContainer" containerID="64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb" Nov 02 22:53:53.870916 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:53.870894 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb\": container with ID starting with 64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb not found: ID does not exist" containerID="64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb" Nov 02 22:53:53.870970 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:53.870926 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb"} err="failed to get container status \"64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb\": rpc error: code = NotFound desc = could not find container \"64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb\": container with ID starting with 64f0c04d4362afa786c3622e651bd5bcaba00834462c95a2d0d8980a3d15dadb not found: ID does not exist" Nov 02 22:53:54.019056 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:54.019026 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f695d65dd-478hr"] Nov 02 22:53:54.065436 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:54.065396 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f695d65dd-478hr"] Nov 02 22:53:54.709986 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:54.709952 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeea33da-f0ef-4912-8ebc-29b48d336448" path="/var/lib/kubelet/pods/aeea33da-f0ef-4912-8ebc-29b48d336448/volumes" Nov 02 22:53:56.007683 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:56.007655 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2wfzr" Nov 02 22:53:56.019232 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:53:56.019201 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:53:59.929530 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:59.929496 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:53:59.930439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:59.930398 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="prometheus" containerID="cri-o://d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134" gracePeriod=600 Nov 02 22:53:59.930581 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:59.930445 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy-web" containerID="cri-o://e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af" gracePeriod=600 Nov 02 22:53:59.930581 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:59.930464 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9" gracePeriod=600 Nov 02 22:53:59.930581 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:59.930493 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="thanos-sidecar" containerID="cri-o://debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407" gracePeriod=600 Nov 02 22:53:59.930729 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:59.930680 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy" containerID="cri-o://cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de" gracePeriod=600 Nov 02 22:53:59.931408 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:53:59.930579 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="config-reloader" containerID="cri-o://51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a" gracePeriod=600 Nov 02 22:54:00.207554 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.207530 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:00.363571 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.363539 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-tls\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.363740 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.363583 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-kube-rbac-proxy\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.363740 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.363626 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-web-config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-web-config\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.363740 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.363661 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-db\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.363740 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.363685 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-grpc-tls\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.363965 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.363850 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-trusted-ca-bundle\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.363965 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.363899 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-config\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.363965 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.363927 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-metrics-client-certs\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364124 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.363965 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-config-out\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364124 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364005 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-serving-certs-ca-bundle\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364124 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364040 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-kubelet-serving-ca-bundle\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364124 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364069 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-metrics-client-ca\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364124 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364102 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-1\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364414 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364131 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-thanos-prometheus-http-client-file\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364414 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364190 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-tls-assets\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-tls-assets\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364414 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364217 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-2\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364414 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364248 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-0\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364414 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364275 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-kube-api-access-qbf65\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-kube-api-access-qbf65\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364414 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364302 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364414 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364337 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f6a9d6b0-dd41-4d54-bf35-110826270d66\" (UID: \"f6a9d6b0-dd41-4d54-bf35-110826270d66\") " Nov 02 22:54:00.364766 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364429 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-trusted-ca-bundle" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:54:00.364766 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364598 5672 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.365176 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.364894 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-db" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:54:00.367175 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.367129 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-grpc-tls" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:00.367175 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.367164 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-2" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:54:00.367394 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.367320 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-kube-rbac-proxy" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:00.367641 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.367620 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-metrics-client-ca" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:54:00.368301 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.368277 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-0" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:54:00.368402 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.368334 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-tls" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:00.368469 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.368442 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-kube-rbac-proxy-web" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:00.368557 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.368519 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-1" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:54:00.368721 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.368515 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-metrics-client-certs" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:00.368721 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.368574 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-thanos-prometheus-http-client-file" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:00.368857 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.368646 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-config" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:00.368914 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.368890 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-tls-assets" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:54:00.368952 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.368930 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-kubelet-serving-ca-bundle" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:54:00.369132 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.369110 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-serving-certs-ca-bundle" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:54:00.369440 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.369420 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-config-out" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:54:00.369541 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.369523 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-kube-api-access-qbf65" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "kube-api-access-qbf65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:54:00.369822 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.369806 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-thanos-sidecar-tls" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:00.379063 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.379037 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-web-config" pod "f6a9d6b0-dd41-4d54-bf35-110826270d66" (UID: "f6a9d6b0-dd41-4d54-bf35-110826270d66"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:00.465570 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465486 5672 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465570 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465517 5672 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465570 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465531 5672 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-tls-assets\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465570 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465545 5672 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465570 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465558 5672 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465570 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465572 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbf65\" (UniqueName: \"kubernetes.io/projected/f6a9d6b0-dd41-4d54-bf35-110826270d66-kube-api-access-qbf65\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465585 5672 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465599 5672 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465613 5672 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465625 5672 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-kube-rbac-proxy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465638 5672 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-web-config\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465651 5672 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-prometheus-k8s-db\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465663 5672 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-grpc-tls\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465676 5672 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-config\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465688 5672 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f6a9d6b0-dd41-4d54-bf35-110826270d66-secret-metrics-client-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465700 5672 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6a9d6b0-dd41-4d54-bf35-110826270d66-config-out\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465713 5672 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465726 5672 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.465902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.465740 5672 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6a9d6b0-dd41-4d54-bf35-110826270d66-configmap-metrics-client-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:00.891062 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891030 5672 generic.go:356] "Generic (PLEG): container finished" podID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerID="e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9" exitCode=0 Nov 02 22:54:00.891062 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891054 5672 generic.go:356] "Generic (PLEG): container finished" podID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerID="cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de" exitCode=0 Nov 02 22:54:00.891062 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891062 5672 generic.go:356] "Generic (PLEG): container finished" podID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerID="e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af" exitCode=0 Nov 02 22:54:00.891062 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891068 5672 generic.go:356] "Generic (PLEG): container finished" podID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerID="debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407" exitCode=0 Nov 02 22:54:00.891333 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891075 5672 generic.go:356] "Generic (PLEG): container finished" podID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerID="51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a" exitCode=0 Nov 02 22:54:00.891333 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891080 5672 generic.go:356] "Generic (PLEG): container finished" podID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerID="d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134" exitCode=0 Nov 02 22:54:00.891333 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891113 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:00.891333 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891110 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerDied","Data":"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9"} Nov 02 22:54:00.891333 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891210 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerDied","Data":"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de"} Nov 02 22:54:00.891333 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891258 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerDied","Data":"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af"} Nov 02 22:54:00.891333 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891273 5672 scope.go:122] "RemoveContainer" containerID="e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9" Nov 02 22:54:00.891333 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891302 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerDied","Data":"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407"} Nov 02 22:54:00.891606 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891344 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerDied","Data":"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a"} Nov 02 22:54:00.891606 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891383 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerDied","Data":"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134"} Nov 02 22:54:00.891606 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.891426 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f6a9d6b0-dd41-4d54-bf35-110826270d66","Type":"ContainerDied","Data":"07fde3b76be6a14be4598fa33edaa6ecf159820c47afd3b0aea63a609037b676"} Nov 02 22:54:00.898640 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.898623 5672 scope.go:122] "RemoveContainer" containerID="cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de" Nov 02 22:54:00.905054 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.905023 5672 scope.go:122] "RemoveContainer" containerID="e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af" Nov 02 22:54:00.911339 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.911320 5672 scope.go:122] "RemoveContainer" containerID="debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407" Nov 02 22:54:00.917242 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.917225 5672 scope.go:122] "RemoveContainer" containerID="51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a" Nov 02 22:54:00.923708 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.923691 5672 scope.go:122] "RemoveContainer" containerID="d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134" Nov 02 22:54:00.930024 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.930005 5672 scope.go:122] "RemoveContainer" containerID="89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c" Nov 02 22:54:00.937861 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.937834 5672 scope.go:122] "RemoveContainer" containerID="e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9" Nov 02 22:54:00.938310 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:54:00.938289 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": container with ID starting with e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9 not found: ID does not exist" containerID="e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9" Nov 02 22:54:00.938398 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.938319 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9"} err="failed to get container status \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": rpc error: code = NotFound desc = could not find container \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": container with ID starting with e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9 not found: ID does not exist" Nov 02 22:54:00.938398 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.938337 5672 scope.go:122] "RemoveContainer" containerID="cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de" Nov 02 22:54:00.938590 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:54:00.938574 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": container with ID starting with cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de not found: ID does not exist" containerID="cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de" Nov 02 22:54:00.938658 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.938595 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de"} err="failed to get container status \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": rpc error: code = NotFound desc = could not find container \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": container with ID starting with cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de not found: ID does not exist" Nov 02 22:54:00.938658 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.938610 5672 scope.go:122] "RemoveContainer" containerID="e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af" Nov 02 22:54:00.938853 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:54:00.938835 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": container with ID starting with e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af not found: ID does not exist" containerID="e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af" Nov 02 22:54:00.938918 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.938863 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af"} err="failed to get container status \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": rpc error: code = NotFound desc = could not find container \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": container with ID starting with e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af not found: ID does not exist" Nov 02 22:54:00.938918 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.938883 5672 scope.go:122] "RemoveContainer" containerID="debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407" Nov 02 22:54:00.939119 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:54:00.939104 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": container with ID starting with debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407 not found: ID does not exist" containerID="debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407" Nov 02 22:54:00.939177 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.939122 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407"} err="failed to get container status \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": rpc error: code = NotFound desc = could not find container \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": container with ID starting with debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407 not found: ID does not exist" Nov 02 22:54:00.939177 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.939135 5672 scope.go:122] "RemoveContainer" containerID="51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a" Nov 02 22:54:00.939434 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:54:00.939416 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": container with ID starting with 51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a not found: ID does not exist" containerID="51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a" Nov 02 22:54:00.939482 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.939440 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a"} err="failed to get container status \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": rpc error: code = NotFound desc = could not find container \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": container with ID starting with 51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a not found: ID does not exist" Nov 02 22:54:00.939482 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.939453 5672 scope.go:122] "RemoveContainer" containerID="d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134" Nov 02 22:54:00.939698 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:54:00.939675 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": container with ID starting with d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134 not found: ID does not exist" containerID="d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134" Nov 02 22:54:00.939777 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.939701 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134"} err="failed to get container status \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": rpc error: code = NotFound desc = could not find container \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": container with ID starting with d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134 not found: ID does not exist" Nov 02 22:54:00.939777 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.939714 5672 scope.go:122] "RemoveContainer" containerID="89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c" Nov 02 22:54:00.939921 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:54:00.939905 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": container with ID starting with 89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c not found: ID does not exist" containerID="89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c" Nov 02 22:54:00.939955 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.939925 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c"} err="failed to get container status \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": rpc error: code = NotFound desc = could not find container \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": container with ID starting with 89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c not found: ID does not exist" Nov 02 22:54:00.939955 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.939937 5672 scope.go:122] "RemoveContainer" containerID="e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9" Nov 02 22:54:00.940143 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.940127 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9"} err="failed to get container status \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": rpc error: code = NotFound desc = could not find container \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": container with ID starting with e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9 not found: ID does not exist" Nov 02 22:54:00.940194 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.940143 5672 scope.go:122] "RemoveContainer" containerID="cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de" Nov 02 22:54:00.940401 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.940373 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de"} err="failed to get container status \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": rpc error: code = NotFound desc = could not find container \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": container with ID starting with cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de not found: ID does not exist" Nov 02 22:54:00.940401 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.940393 5672 scope.go:122] "RemoveContainer" containerID="e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af" Nov 02 22:54:00.940653 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.940638 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af"} err="failed to get container status \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": rpc error: code = NotFound desc = could not find container \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": container with ID starting with e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af not found: ID does not exist" Nov 02 22:54:00.940701 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.940653 5672 scope.go:122] "RemoveContainer" containerID="debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407" Nov 02 22:54:00.940849 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.940826 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407"} err="failed to get container status \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": rpc error: code = NotFound desc = could not find container \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": container with ID starting with debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407 not found: ID does not exist" Nov 02 22:54:00.940895 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.940851 5672 scope.go:122] "RemoveContainer" containerID="51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a" Nov 02 22:54:00.941095 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.941073 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a"} err="failed to get container status \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": rpc error: code = NotFound desc = could not find container \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": container with ID starting with 51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a not found: ID does not exist" Nov 02 22:54:00.941171 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.941095 5672 scope.go:122] "RemoveContainer" containerID="d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134" Nov 02 22:54:00.941338 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.941319 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134"} err="failed to get container status \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": rpc error: code = NotFound desc = could not find container \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": container with ID starting with d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134 not found: ID does not exist" Nov 02 22:54:00.941397 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.941339 5672 scope.go:122] "RemoveContainer" containerID="89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c" Nov 02 22:54:00.941552 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.941536 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c"} err="failed to get container status \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": rpc error: code = NotFound desc = could not find container \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": container with ID starting with 89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c not found: ID does not exist" Nov 02 22:54:00.941602 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.941552 5672 scope.go:122] "RemoveContainer" containerID="e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9" Nov 02 22:54:00.941753 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.941738 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9"} err="failed to get container status \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": rpc error: code = NotFound desc = could not find container \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": container with ID starting with e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9 not found: ID does not exist" Nov 02 22:54:00.941753 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.941752 5672 scope.go:122] "RemoveContainer" containerID="cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de" Nov 02 22:54:00.941973 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.941954 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de"} err="failed to get container status \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": rpc error: code = NotFound desc = could not find container \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": container with ID starting with cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de not found: ID does not exist" Nov 02 22:54:00.942014 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.941974 5672 scope.go:122] "RemoveContainer" containerID="e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af" Nov 02 22:54:00.942238 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.942215 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af"} err="failed to get container status \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": rpc error: code = NotFound desc = could not find container \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": container with ID starting with e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af not found: ID does not exist" Nov 02 22:54:00.942238 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.942237 5672 scope.go:122] "RemoveContainer" containerID="debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407" Nov 02 22:54:00.942465 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.942447 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407"} err="failed to get container status \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": rpc error: code = NotFound desc = could not find container \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": container with ID starting with debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407 not found: ID does not exist" Nov 02 22:54:00.942540 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.942467 5672 scope.go:122] "RemoveContainer" containerID="51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a" Nov 02 22:54:00.942700 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.942683 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a"} err="failed to get container status \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": rpc error: code = NotFound desc = could not find container \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": container with ID starting with 51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a not found: ID does not exist" Nov 02 22:54:00.942737 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.942700 5672 scope.go:122] "RemoveContainer" containerID="d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134" Nov 02 22:54:00.942907 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.942890 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134"} err="failed to get container status \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": rpc error: code = NotFound desc = could not find container \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": container with ID starting with d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134 not found: ID does not exist" Nov 02 22:54:00.942976 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.942908 5672 scope.go:122] "RemoveContainer" containerID="89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c" Nov 02 22:54:00.943114 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.943098 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c"} err="failed to get container status \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": rpc error: code = NotFound desc = could not find container \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": container with ID starting with 89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c not found: ID does not exist" Nov 02 22:54:00.943175 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.943115 5672 scope.go:122] "RemoveContainer" containerID="e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9" Nov 02 22:54:00.943310 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.943291 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9"} err="failed to get container status \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": rpc error: code = NotFound desc = could not find container \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": container with ID starting with e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9 not found: ID does not exist" Nov 02 22:54:00.943354 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.943311 5672 scope.go:122] "RemoveContainer" containerID="cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de" Nov 02 22:54:00.943499 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.943481 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de"} err="failed to get container status \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": rpc error: code = NotFound desc = could not find container \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": container with ID starting with cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de not found: ID does not exist" Nov 02 22:54:00.943568 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.943499 5672 scope.go:122] "RemoveContainer" containerID="e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af" Nov 02 22:54:00.943700 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.943682 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af"} err="failed to get container status \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": rpc error: code = NotFound desc = could not find container \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": container with ID starting with e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af not found: ID does not exist" Nov 02 22:54:00.943745 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.943700 5672 scope.go:122] "RemoveContainer" containerID="debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407" Nov 02 22:54:00.943901 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.943884 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407"} err="failed to get container status \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": rpc error: code = NotFound desc = could not find container \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": container with ID starting with debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407 not found: ID does not exist" Nov 02 22:54:00.943965 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.943902 5672 scope.go:122] "RemoveContainer" containerID="51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a" Nov 02 22:54:00.944116 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.944102 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a"} err="failed to get container status \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": rpc error: code = NotFound desc = could not find container \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": container with ID starting with 51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a not found: ID does not exist" Nov 02 22:54:00.944189 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.944117 5672 scope.go:122] "RemoveContainer" containerID="d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134" Nov 02 22:54:00.944334 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.944317 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134"} err="failed to get container status \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": rpc error: code = NotFound desc = could not find container \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": container with ID starting with d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134 not found: ID does not exist" Nov 02 22:54:00.944395 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.944334 5672 scope.go:122] "RemoveContainer" containerID="89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c" Nov 02 22:54:00.944526 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.944507 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c"} err="failed to get container status \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": rpc error: code = NotFound desc = could not find container \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": container with ID starting with 89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c not found: ID does not exist" Nov 02 22:54:00.944572 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.944527 5672 scope.go:122] "RemoveContainer" containerID="e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9" Nov 02 22:54:00.944699 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.944681 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9"} err="failed to get container status \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": rpc error: code = NotFound desc = could not find container \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": container with ID starting with e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9 not found: ID does not exist" Nov 02 22:54:00.944763 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.944699 5672 scope.go:122] "RemoveContainer" containerID="cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de" Nov 02 22:54:00.944887 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.944872 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de"} err="failed to get container status \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": rpc error: code = NotFound desc = could not find container \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": container with ID starting with cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de not found: ID does not exist" Nov 02 22:54:00.944954 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.944888 5672 scope.go:122] "RemoveContainer" containerID="e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af" Nov 02 22:54:00.945082 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.945062 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af"} err="failed to get container status \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": rpc error: code = NotFound desc = could not find container \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": container with ID starting with e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af not found: ID does not exist" Nov 02 22:54:00.945082 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.945080 5672 scope.go:122] "RemoveContainer" containerID="debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407" Nov 02 22:54:00.945272 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.945256 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407"} err="failed to get container status \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": rpc error: code = NotFound desc = could not find container \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": container with ID starting with debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407 not found: ID does not exist" Nov 02 22:54:00.945314 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.945272 5672 scope.go:122] "RemoveContainer" containerID="51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a" Nov 02 22:54:00.945490 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.945472 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a"} err="failed to get container status \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": rpc error: code = NotFound desc = could not find container \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": container with ID starting with 51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a not found: ID does not exist" Nov 02 22:54:00.945605 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.945490 5672 scope.go:122] "RemoveContainer" containerID="d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134" Nov 02 22:54:00.946233 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.946200 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134"} err="failed to get container status \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": rpc error: code = NotFound desc = could not find container \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": container with ID starting with d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134 not found: ID does not exist" Nov 02 22:54:00.946233 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.946232 5672 scope.go:122] "RemoveContainer" containerID="89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c" Nov 02 22:54:00.949801 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.949776 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c"} err="failed to get container status \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": rpc error: code = NotFound desc = could not find container \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": container with ID starting with 89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c not found: ID does not exist" Nov 02 22:54:00.949801 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.949801 5672 scope.go:122] "RemoveContainer" containerID="e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9" Nov 02 22:54:00.950033 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.950015 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9"} err="failed to get container status \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": rpc error: code = NotFound desc = could not find container \"e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9\": container with ID starting with e36b6e1c6a2ca21944372bc14e61354e9d74bec65be7bc28a79a2ea8d09eeca9 not found: ID does not exist" Nov 02 22:54:00.950105 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.950034 5672 scope.go:122] "RemoveContainer" containerID="cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de" Nov 02 22:54:00.950291 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.950272 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de"} err="failed to get container status \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": rpc error: code = NotFound desc = could not find container \"cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de\": container with ID starting with cfd7489281a7a7f3917c9992da1945af82f870064e5830a9c78bde78869d97de not found: ID does not exist" Nov 02 22:54:00.950356 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.950291 5672 scope.go:122] "RemoveContainer" containerID="e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af" Nov 02 22:54:00.950568 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.950546 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af"} err="failed to get container status \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": rpc error: code = NotFound desc = could not find container \"e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af\": container with ID starting with e5155c97a13eed043e0d0dbdfa4ef4641b664856c2a9ffc5e39ed966fd8e44af not found: ID does not exist" Nov 02 22:54:00.950638 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.950569 5672 scope.go:122] "RemoveContainer" containerID="debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407" Nov 02 22:54:00.950807 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.950787 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407"} err="failed to get container status \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": rpc error: code = NotFound desc = could not find container \"debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407\": container with ID starting with debaa2ea638306a62eb012af6f2181b6c0ffa0d27b378c62f60bf23934bcf407 not found: ID does not exist" Nov 02 22:54:00.950852 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.950808 5672 scope.go:122] "RemoveContainer" containerID="51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a" Nov 02 22:54:00.951078 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.951045 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a"} err="failed to get container status \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": rpc error: code = NotFound desc = could not find container \"51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a\": container with ID starting with 51c54de78d10d6dc4db042d1d6ed5e8a99e1814dc130711b99c996201cf1816a not found: ID does not exist" Nov 02 22:54:00.951116 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.951085 5672 scope.go:122] "RemoveContainer" containerID="d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134" Nov 02 22:54:00.951453 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.951421 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134"} err="failed to get container status \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": rpc error: code = NotFound desc = could not find container \"d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134\": container with ID starting with d188fa75e791bef80b7c8672bc5892688015290d3927e867dc2a1fa2d895e134 not found: ID does not exist" Nov 02 22:54:00.951529 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.951458 5672 scope.go:122] "RemoveContainer" containerID="89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c" Nov 02 22:54:00.951832 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:00.951814 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c"} err="failed to get container status \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": rpc error: code = NotFound desc = could not find container \"89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c\": container with ID starting with 89b5b44d1b467fb74c033d0bcfb3135f0d26ffe2172dfad3b4317a4911cb813c not found: ID does not exist" Nov 02 22:54:01.048613 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.048589 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:54:01.095117 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095086 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:54:01.095322 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095306 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:54:01.095585 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095574 5672 cpu_manager.go:414] "containerMap: removing container" podUID="aeea33da-f0ef-4912-8ebc-29b48d336448" containerName="console" Nov 02 22:54:01.095616 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095587 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeea33da-f0ef-4912-8ebc-29b48d336448" containerName="console" Nov 02 22:54:01.095616 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095601 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy-thanos" Nov 02 22:54:01.095616 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095606 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy-thanos" Nov 02 22:54:01.095616 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095613 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="thanos-sidecar" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095618 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="thanos-sidecar" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095629 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy-web" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095634 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy-web" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095641 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="init-config-reloader" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095645 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="init-config-reloader" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095652 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095657 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095663 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="prometheus" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095667 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="prometheus" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095678 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="config-reloader" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095682 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="config-reloader" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095721 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeea33da-f0ef-4912-8ebc-29b48d336448" containerName="console" Nov 02 22:54:01.095727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095729 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy-thanos" Nov 02 22:54:01.096068 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095735 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="thanos-sidecar" Nov 02 22:54:01.096068 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095742 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy-web" Nov 02 22:54:01.096068 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095748 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="init-config-reloader" Nov 02 22:54:01.096068 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095753 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="kube-rbac-proxy" Nov 02 22:54:01.096068 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095757 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="prometheus" Nov 02 22:54:01.096068 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.095763 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" containerName="config-reloader" Nov 02 22:54:01.130636 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.130615 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.181245 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.181214 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 22:54:01.181348 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.181257 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 22:54:01.181464 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.181448 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 22:54:01.181510 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.181490 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 22:54:01.182004 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.181985 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 22:54:01.182126 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.182023 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 22:54:01.182126 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.182022 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 22:54:01.182126 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.182040 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 22:54:01.182126 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.182027 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 22:54:01.182126 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.182065 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 22:54:01.182126 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.182073 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 22:54:01.182126 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.182087 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-75wn4\"" Nov 02 22:54:01.182425 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.182172 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 22:54:01.182425 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.182185 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-tedh3m60u5r0\"" Nov 02 22:54:01.185694 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.185679 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 22:54:01.233443 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.233413 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 22:54:01.233443 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.233437 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 22:54:01.233890 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.233874 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:54:01.271523 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.271502 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f5f0355-698a-4964-a644-68568dbbfe33-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.271634 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.271530 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.271634 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.271547 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.271634 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.271567 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-config\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.271795 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.271714 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f5f0355-698a-4964-a644-68568dbbfe33-config-out\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.271795 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.271766 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.271888 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.271868 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.271923 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.271904 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.271959 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.271934 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.271990 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.271965 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.272036 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.271990 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.272036 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.272024 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.272098 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.272050 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-web-config\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.272098 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.272066 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.272172 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.272102 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgrt5\" (UniqueName: \"kubernetes.io/projected/9f5f0355-698a-4964-a644-68568dbbfe33-kube-api-access-mgrt5\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.272172 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.272127 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.272246 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.272194 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.272246 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.272224 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.272301 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.272259 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.272301 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.272279 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.373460 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373404 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f5f0355-698a-4964-a644-68568dbbfe33-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.373460 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373434 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.373460 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373452 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.373670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373483 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-config\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.373670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373512 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f5f0355-698a-4964-a644-68568dbbfe33-config-out\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.373670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373535 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.373670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373575 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.373670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373601 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.373670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373624 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.373670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373649 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.373670 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373671 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374056 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373699 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374056 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373722 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-web-config\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374056 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373747 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374056 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373770 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrt5\" (UniqueName: \"kubernetes.io/projected/9f5f0355-698a-4964-a644-68568dbbfe33-kube-api-access-mgrt5\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374056 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373793 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374056 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373824 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374056 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373838 5672 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:54:01.374056 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373852 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374056 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373886 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374056 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.373913 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374583 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.374128 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374583 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.374126 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374583 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.374390 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374583 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.374497 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.374853 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.374826 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.376571 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.376542 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f5f0355-698a-4964-a644-68568dbbfe33-config-out\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.376571 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.376561 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f5f0355-698a-4964-a644-68568dbbfe33-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.377487 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.376949 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-config\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.377487 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.377211 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.377487 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.377229 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-web-config\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.377487 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.377233 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.377487 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.377401 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.377793 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.377675 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.377793 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.377676 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.378060 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.378036 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.379169 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.379122 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.379247 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.379207 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.379800 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.379783 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9f5f0355-698a-4964-a644-68568dbbfe33-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.380971 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.380949 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9f5f0355-698a-4964-a644-68568dbbfe33-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.430269 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.430242 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrt5\" (UniqueName: \"kubernetes.io/projected/9f5f0355-698a-4964-a644-68568dbbfe33-kube-api-access-mgrt5\") pod \"prometheus-k8s-0\" (UID: \"9f5f0355-698a-4964-a644-68568dbbfe33\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.533545 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.533508 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:01.992298 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:01.992269 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:54:01.995167 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:54:01.995120 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5f0355_698a_4964_a644_68568dbbfe33.slice/crio-ee5c9fc5042ea1cba9c34efe5204844e993aeddc419e3caf74f6ed6e3b4851de WatchSource:0}: Error finding container ee5c9fc5042ea1cba9c34efe5204844e993aeddc419e3caf74f6ed6e3b4851de: Status 404 returned error can't find the container with id ee5c9fc5042ea1cba9c34efe5204844e993aeddc419e3caf74f6ed6e3b4851de Nov 02 22:54:02.710407 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:02.710377 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a9d6b0-dd41-4d54-bf35-110826270d66" path="/var/lib/kubelet/pods/f6a9d6b0-dd41-4d54-bf35-110826270d66/volumes" Nov 02 22:54:02.900410 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:02.900382 5672 generic.go:356] "Generic (PLEG): container finished" podID="9f5f0355-698a-4964-a644-68568dbbfe33" containerID="c569eb636c1213826b343bbbfb5666e77cb258000755e512650533a7d3cff49c" exitCode=0 Nov 02 22:54:02.900586 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:02.900419 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f5f0355-698a-4964-a644-68568dbbfe33","Type":"ContainerDied","Data":"c569eb636c1213826b343bbbfb5666e77cb258000755e512650533a7d3cff49c"} Nov 02 22:54:02.900586 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:02.900467 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f5f0355-698a-4964-a644-68568dbbfe33","Type":"ContainerStarted","Data":"ee5c9fc5042ea1cba9c34efe5204844e993aeddc419e3caf74f6ed6e3b4851de"} Nov 02 22:54:03.907577 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:03.907544 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f5f0355-698a-4964-a644-68568dbbfe33","Type":"ContainerStarted","Data":"783c64f50d2529e100ad123f1f59c271e6e873a88161ea3ba9574f80c6734467"} Nov 02 22:54:03.908087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:03.907612 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f5f0355-698a-4964-a644-68568dbbfe33","Type":"ContainerStarted","Data":"0838c1dd10773cc0373b9287893debde744632923f7419618615aa2f32abec35"} Nov 02 22:54:03.908087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:03.907641 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f5f0355-698a-4964-a644-68568dbbfe33","Type":"ContainerStarted","Data":"95b3f257d8777cdd222cfd450ea28b6837d8a721e4a5fa74b8466ae2e85f4533"} Nov 02 22:54:03.908087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:03.907664 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f5f0355-698a-4964-a644-68568dbbfe33","Type":"ContainerStarted","Data":"0c99c8ec11433c2540563197a36118db7e3b60bd8b6c6623fb4305571087180d"} Nov 02 22:54:03.908087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:03.907691 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f5f0355-698a-4964-a644-68568dbbfe33","Type":"ContainerStarted","Data":"469e256b46805db72f733087907ed8f0eaa36b87bde15384b0cb35b88f58de72"} Nov 02 22:54:03.908087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:03.907723 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9f5f0355-698a-4964-a644-68568dbbfe33","Type":"ContainerStarted","Data":"4fcd1897cf8be303ee90d9aa68527108a4d6218067240faf2b01a89ad776447e"} Nov 02 22:54:04.024637 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:04.024588 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.004144831 podStartE2EDuration="3.024570456s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.425625ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:04.02352488 +0000 UTC m=+37.430836709" watchObservedRunningTime="2026-11-02 22:54:04.024570456 +0000 UTC m=+37.431882328" Nov 02 22:54:06.534319 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:06.534276 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:17.102626 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:17.102588 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:17.117777 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:17.117752 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:17.975293 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:54:17.975266 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:54:26.018716 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:54:26.018676 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:54:56.019083 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:54:56.019038 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:55:26.018735 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:55:26.018654 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:55:56.019310 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:55:56.019267 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:56:05.157040 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.157009 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn"] Nov 02 22:56:05.159537 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.159521 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:05.208134 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.208115 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 22:56:05.254421 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.254395 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-g6dh2\"" Nov 02 22:56:05.254536 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.254435 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 22:56:05.254975 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.254955 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn"] Nov 02 22:56:05.257718 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.257701 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:05.257797 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.257731 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmkn\" (UniqueName: \"kubernetes.io/projected/e5c59539-2cdd-434b-ba77-e3585b8df55c-kube-api-access-rhmkn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:05.257797 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.257766 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:05.358526 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.358501 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:05.358635 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.358552 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:05.358635 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.358574 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmkn\" (UniqueName: \"kubernetes.io/projected/e5c59539-2cdd-434b-ba77-e3585b8df55c-kube-api-access-rhmkn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:05.358852 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.358836 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:05.358891 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.358870 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:05.411981 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.411929 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmkn\" (UniqueName: \"kubernetes.io/projected/e5c59539-2cdd-434b-ba77-e3585b8df55c-kube-api-access-rhmkn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:05.472790 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.472764 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:05.830847 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.830817 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn"] Nov 02 22:56:05.834133 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:56:05.834109 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c59539_2cdd_434b_ba77_e3585b8df55c.slice/crio-e935d89ce2ee3a8c43d7e0c0448f4eb6079e4724d190ba23829d7ac1b240fcdb WatchSource:0}: Error finding container e935d89ce2ee3a8c43d7e0c0448f4eb6079e4724d190ba23829d7ac1b240fcdb: Status 404 returned error can't find the container with id e935d89ce2ee3a8c43d7e0c0448f4eb6079e4724d190ba23829d7ac1b240fcdb Nov 02 22:56:05.836388 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.836370 5672 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:56:05.903557 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.903531 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd"] Nov 02 22:56:05.906562 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.906546 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:05.954751 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.954723 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-ww42x\"" Nov 02 22:56:05.960111 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.960086 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd"] Nov 02 22:56:05.963047 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.963016 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:05.963178 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.963120 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:05.963243 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.963178 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8dd\" (UniqueName: \"kubernetes.io/projected/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-kube-api-access-bp8dd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:05.997737 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:05.997698 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m"] Nov 02 22:56:06.000203 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.000188 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:06.048644 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.048620 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-wtn6g\"" Nov 02 22:56:06.054073 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.054052 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m"] Nov 02 22:56:06.064484 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.064461 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:06.064592 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.064492 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:06.064592 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.064516 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-786xf\" (UniqueName: \"kubernetes.io/projected/510980f2-e363-4b29-8ee2-50f66965eb41-kube-api-access-786xf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:06.064728 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.064708 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:06.064775 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.064758 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:06.064817 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.064796 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8dd\" (UniqueName: \"kubernetes.io/projected/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-kube-api-access-bp8dd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:06.064912 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.064896 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:06.064978 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.064960 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:06.099827 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.099777 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l"] Nov 02 22:56:06.102126 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.102111 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:06.119771 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.119749 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8dd\" (UniqueName: \"kubernetes.io/projected/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-kube-api-access-bp8dd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:06.150988 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.150970 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-t6wmv\"" Nov 02 22:56:06.156368 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.156337 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l"] Nov 02 22:56:06.166229 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.166205 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:06.166561 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.166273 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:06.166561 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.166328 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5j8w\" (UniqueName: \"kubernetes.io/projected/ef4c2feb-354b-449e-85ef-c503c66603f8-kube-api-access-n5j8w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:06.166561 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.166386 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:06.166561 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.166446 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-786xf\" (UniqueName: \"kubernetes.io/projected/510980f2-e363-4b29-8ee2-50f66965eb41-kube-api-access-786xf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:06.166561 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.166476 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:06.166738 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.166566 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:06.166738 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.166636 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:06.198995 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.198971 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98"] Nov 02 22:56:06.201207 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.201194 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:06.219100 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.219081 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:06.226464 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.226442 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-786xf\" (UniqueName: \"kubernetes.io/projected/510980f2-e363-4b29-8ee2-50f66965eb41-kube-api-access-786xf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:06.250978 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.250953 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-2hp8d\"" Nov 02 22:56:06.256961 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.256887 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98"] Nov 02 22:56:06.267887 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.267756 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwqzw\" (UniqueName: \"kubernetes.io/projected/bee85a62-333f-4a03-8ef4-2908ebf1e990-kube-api-access-dwqzw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:06.267887 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.267804 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:06.268011 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.267925 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:06.268011 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.267978 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:06.268162 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.268045 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:06.268162 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.268100 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5j8w\" (UniqueName: \"kubernetes.io/projected/ef4c2feb-354b-449e-85ef-c503c66603f8-kube-api-access-n5j8w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:06.268752 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.268731 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:06.268974 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.268955 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:06.304308 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.304263 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" event={"ID":"e5c59539-2cdd-434b-ba77-e3585b8df55c","Type":"ContainerStarted","Data":"e935d89ce2ee3a8c43d7e0c0448f4eb6079e4724d190ba23829d7ac1b240fcdb"} Nov 02 22:56:06.313420 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.313406 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:06.323057 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.323036 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5j8w\" (UniqueName: \"kubernetes.io/projected/ef4c2feb-354b-449e-85ef-c503c66603f8-kube-api-access-n5j8w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:06.368778 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.368745 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwqzw\" (UniqueName: \"kubernetes.io/projected/bee85a62-333f-4a03-8ef4-2908ebf1e990-kube-api-access-dwqzw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:06.368881 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.368800 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:06.368881 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.368862 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:06.369202 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.369185 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:06.369258 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.369240 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:06.415373 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.415344 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:06.422327 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.422302 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwqzw\" (UniqueName: \"kubernetes.io/projected/bee85a62-333f-4a03-8ef4-2908ebf1e990-kube-api-access-dwqzw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:06.513880 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.513854 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:06.578973 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.578949 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd"] Nov 02 22:56:06.583209 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:56:06.583184 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63f6bbe_df96_4b1d_9da9_043ed361ca1d.slice/crio-06f83c14778f4deab4718e0a0f2df03dd5afc0b739e76b498dbb33b59088c94a WatchSource:0}: Error finding container 06f83c14778f4deab4718e0a0f2df03dd5afc0b739e76b498dbb33b59088c94a: Status 404 returned error can't find the container with id 06f83c14778f4deab4718e0a0f2df03dd5afc0b739e76b498dbb33b59088c94a Nov 02 22:56:06.671287 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.671221 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m"] Nov 02 22:56:06.673866 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:56:06.673840 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510980f2_e363_4b29_8ee2_50f66965eb41.slice/crio-2042b676b3a8b65ee52e3c818338898c124fd6b58f66a59186484b6d051a498e WatchSource:0}: Error finding container 2042b676b3a8b65ee52e3c818338898c124fd6b58f66a59186484b6d051a498e: Status 404 returned error can't find the container with id 2042b676b3a8b65ee52e3c818338898c124fd6b58f66a59186484b6d051a498e Nov 02 22:56:06.773950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.773916 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l"] Nov 02 22:56:06.777092 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:56:06.777060 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4c2feb_354b_449e_85ef_c503c66603f8.slice/crio-3d896fa5bcdd0a84c8af7b59895763ff45727163286779fab63217561385a0ba WatchSource:0}: Error finding container 3d896fa5bcdd0a84c8af7b59895763ff45727163286779fab63217561385a0ba: Status 404 returned error can't find the container with id 3d896fa5bcdd0a84c8af7b59895763ff45727163286779fab63217561385a0ba Nov 02 22:56:06.871345 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:06.871318 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98"] Nov 02 22:56:06.874438 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:56:06.874412 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee85a62_333f_4a03_8ef4_2908ebf1e990.slice/crio-4a8d58d83b167d4737634973102a2cd779dbbffb39b6aef7ab6821af410abee0 WatchSource:0}: Error finding container 4a8d58d83b167d4737634973102a2cd779dbbffb39b6aef7ab6821af410abee0: Status 404 returned error can't find the container with id 4a8d58d83b167d4737634973102a2cd779dbbffb39b6aef7ab6821af410abee0 Nov 02 22:56:07.310383 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:07.310344 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" event={"ID":"a63f6bbe-df96-4b1d-9da9-043ed361ca1d","Type":"ContainerStarted","Data":"06f83c14778f4deab4718e0a0f2df03dd5afc0b739e76b498dbb33b59088c94a"} Nov 02 22:56:07.311664 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:07.311630 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" event={"ID":"bee85a62-333f-4a03-8ef4-2908ebf1e990","Type":"ContainerStarted","Data":"4a8d58d83b167d4737634973102a2cd779dbbffb39b6aef7ab6821af410abee0"} Nov 02 22:56:07.312959 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:07.312937 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" event={"ID":"ef4c2feb-354b-449e-85ef-c503c66603f8","Type":"ContainerStarted","Data":"3d896fa5bcdd0a84c8af7b59895763ff45727163286779fab63217561385a0ba"} Nov 02 22:56:07.314230 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:07.314204 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" event={"ID":"510980f2-e363-4b29-8ee2-50f66965eb41","Type":"ContainerStarted","Data":"2042b676b3a8b65ee52e3c818338898c124fd6b58f66a59186484b6d051a498e"} Nov 02 22:56:08.221353 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.221318 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g"] Nov 02 22:56:08.224521 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.224496 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:08.284360 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.284322 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:08.284583 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.284386 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:08.284583 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.284437 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqtw\" (UniqueName: \"kubernetes.io/projected/bf15ddf5-c075-4c71-9ab2-5e01a9637021-kube-api-access-4dqtw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:08.321704 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.321434 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g"] Nov 02 22:56:08.321704 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.321613 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-szr7t\"" Nov 02 22:56:08.386009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.384875 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:08.386009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.384921 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:08.386009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.385022 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqtw\" (UniqueName: \"kubernetes.io/projected/bf15ddf5-c075-4c71-9ab2-5e01a9637021-kube-api-access-4dqtw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:08.386009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.385692 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:08.386009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.385957 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:08.443284 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.443167 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqtw\" (UniqueName: \"kubernetes.io/projected/bf15ddf5-c075-4c71-9ab2-5e01a9637021-kube-api-access-4dqtw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:08.544212 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.544106 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:08.945893 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.944010 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt"] Nov 02 22:56:08.960350 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.947757 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:08.989503 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.989476 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g"] Nov 02 22:56:08.990414 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.990389 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:08.990533 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.990428 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:08.990533 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:08.990452 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp9w2\" (UniqueName: \"kubernetes.io/projected/820f8bff-cae1-448c-8c26-2b40775cba71-kube-api-access-jp9w2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:09.000491 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.000464 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt"] Nov 02 22:56:09.002734 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.002713 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-bw6kf\"" Nov 02 22:56:09.091276 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.091219 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:09.091463 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.091311 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:09.091463 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.091350 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp9w2\" (UniqueName: \"kubernetes.io/projected/820f8bff-cae1-448c-8c26-2b40775cba71-kube-api-access-jp9w2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:09.092167 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.092103 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:09.092559 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.092484 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:09.149042 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.148980 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp9w2\" (UniqueName: \"kubernetes.io/projected/820f8bff-cae1-448c-8c26-2b40775cba71-kube-api-access-jp9w2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:09.278917 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.277694 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:09.329795 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.329760 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" event={"ID":"bf15ddf5-c075-4c71-9ab2-5e01a9637021","Type":"ContainerStarted","Data":"8c93d34715cced77988090d043d0c51c5a8a045ef0577a2f443a4238952a5af8"} Nov 02 22:56:09.571996 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.570742 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6"] Nov 02 22:56:09.574963 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.573927 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.623179 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.623101 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:56:09.623179 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.623101 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 22:56:09.623387 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.623261 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tbbgz\"" Nov 02 22:56:09.646921 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.646873 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 22:56:09.670857 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.670695 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6"] Nov 02 22:56:09.680276 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.680250 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt"] Nov 02 22:56:09.687217 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:56:09.687187 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820f8bff_cae1_448c_8c26_2b40775cba71.slice/crio-24efbd8f531cd4f6997b847138ea86fac0894a99c5935fdcaaf26d233e2f994f WatchSource:0}: Error finding container 24efbd8f531cd4f6997b847138ea86fac0894a99c5935fdcaaf26d233e2f994f: Status 404 returned error can't find the container with id 24efbd8f531cd4f6997b847138ea86fac0894a99c5935fdcaaf26d233e2f994f Nov 02 22:56:09.697061 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.697034 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm447\" (UniqueName: \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-kube-api-access-nm447\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.697164 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.697082 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.697164 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.697114 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.697294 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.697177 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.697294 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.697206 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.697294 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.697242 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.697294 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.697282 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.697424 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.697308 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.697424 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.697363 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.798087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.798050 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.798313 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.798103 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.798313 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.798141 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.798313 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.798208 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.798313 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.798269 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.798558 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.798339 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.798558 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.798385 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm447\" (UniqueName: \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-kube-api-access-nm447\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.798558 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.798415 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.798558 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.798441 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.799457 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.799420 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.800161 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.800120 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.800932 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.800904 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.801199 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.801136 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.801302 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.799139 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.807744 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.807667 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.810272 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.810229 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.863044 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.862967 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm447\" (UniqueName: \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-kube-api-access-nm447\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.872103 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.872072 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7jln6\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:09.890892 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:09.890863 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:10.337456 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:10.337423 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6"] Nov 02 22:56:10.348496 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:56:10.348467 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd55d7c_49eb_40d2_9daf_49dd2996bc6b.slice/crio-371a3ca8c9c1f56856180ffbad183337113b1c8b8b13a9939e491d839c9d9ee5 WatchSource:0}: Error finding container 371a3ca8c9c1f56856180ffbad183337113b1c8b8b13a9939e491d839c9d9ee5: Status 404 returned error can't find the container with id 371a3ca8c9c1f56856180ffbad183337113b1c8b8b13a9939e491d839c9d9ee5 Nov 02 22:56:10.357023 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:10.356993 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" event={"ID":"820f8bff-cae1-448c-8c26-2b40775cba71","Type":"ContainerStarted","Data":"24efbd8f531cd4f6997b847138ea86fac0894a99c5935fdcaaf26d233e2f994f"} Nov 02 22:56:11.365183 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:11.365123 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" event={"ID":"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b","Type":"ContainerStarted","Data":"371a3ca8c9c1f56856180ffbad183337113b1c8b8b13a9939e491d839c9d9ee5"} Nov 02 22:56:19.588363 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:19.588239 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:56:19.589432 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:19.589254 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:56:19.589432 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:19.589321 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:56:20.444705 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:20.444653 5672 generic.go:356] "Generic (PLEG): container finished" podID="820f8bff-cae1-448c-8c26-2b40775cba71" containerID="b58a4b078d2194470f99294fe12e1eda6fab29ec0903d0f5d9dcd99c588385e2" exitCode=0 Nov 02 22:56:20.444870 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:20.444748 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" event={"ID":"820f8bff-cae1-448c-8c26-2b40775cba71","Type":"ContainerDied","Data":"b58a4b078d2194470f99294fe12e1eda6fab29ec0903d0f5d9dcd99c588385e2"} Nov 02 22:56:20.448404 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:20.448374 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" event={"ID":"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b","Type":"ContainerStarted","Data":"3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727"} Nov 02 22:56:20.451111 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:20.451086 5672 generic.go:356] "Generic (PLEG): container finished" podID="e5c59539-2cdd-434b-ba77-e3585b8df55c" containerID="7d5d955cced7453b513dfc8032f176a1527eb46a7b35089781eb55e0c4989dca" exitCode=0 Nov 02 22:56:20.451213 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:20.451170 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" event={"ID":"e5c59539-2cdd-434b-ba77-e3585b8df55c","Type":"ContainerDied","Data":"7d5d955cced7453b513dfc8032f176a1527eb46a7b35089781eb55e0c4989dca"} Nov 02 22:56:20.769110 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:20.769053 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" podStartSLOduration=2.537970242 podStartE2EDuration="11.769038597s" totalImagesPullingTime="9.231068355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:20.661624807 +0000 UTC m=+174.068936635" watchObservedRunningTime="2026-11-02 22:56:20.769038597 +0000 UTC m=+174.176350425" Nov 02 22:56:20.891386 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:20.891346 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:20.896801 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:20.896774 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:21.457272 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:21.457232 5672 generic.go:356] "Generic (PLEG): container finished" podID="a63f6bbe-df96-4b1d-9da9-043ed361ca1d" containerID="ff70aef5ba084c25d718431ca0bf43974cc2a6603d3efeb39215c32b8371a713" exitCode=0 Nov 02 22:56:21.457449 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:21.457327 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" event={"ID":"a63f6bbe-df96-4b1d-9da9-043ed361ca1d","Type":"ContainerDied","Data":"ff70aef5ba084c25d718431ca0bf43974cc2a6603d3efeb39215c32b8371a713"} Nov 02 22:56:21.459089 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:21.459067 5672 generic.go:356] "Generic (PLEG): container finished" podID="510980f2-e363-4b29-8ee2-50f66965eb41" containerID="06c21c4de48dee45a905412ad9439a70530d796f0ce7d259fbd78da198c87fdc" exitCode=0 Nov 02 22:56:21.459206 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:21.459169 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" event={"ID":"510980f2-e363-4b29-8ee2-50f66965eb41","Type":"ContainerDied","Data":"06c21c4de48dee45a905412ad9439a70530d796f0ce7d259fbd78da198c87fdc"} Nov 02 22:56:21.459483 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:21.459469 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:21.460535 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:21.460518 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:56:22.465960 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:22.465930 5672 generic.go:356] "Generic (PLEG): container finished" podID="bee85a62-333f-4a03-8ef4-2908ebf1e990" containerID="7db7137e93dc7fad08f44631d55fcff99961ee6e9ee0f227fdd4ba3d9d65471b" exitCode=0 Nov 02 22:56:22.466408 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:22.466026 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" event={"ID":"bee85a62-333f-4a03-8ef4-2908ebf1e990","Type":"ContainerDied","Data":"7db7137e93dc7fad08f44631d55fcff99961ee6e9ee0f227fdd4ba3d9d65471b"} Nov 02 22:56:22.470446 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:22.470418 5672 generic.go:356] "Generic (PLEG): container finished" podID="ef4c2feb-354b-449e-85ef-c503c66603f8" containerID="718ef9e8c74311476c7f8e83b049e74bb34876e49d37f36f81866973fd0c1421" exitCode=0 Nov 02 22:56:22.471055 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:22.471037 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" event={"ID":"ef4c2feb-354b-449e-85ef-c503c66603f8","Type":"ContainerDied","Data":"718ef9e8c74311476c7f8e83b049e74bb34876e49d37f36f81866973fd0c1421"} Nov 02 22:56:23.483286 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:23.483227 5672 generic.go:356] "Generic (PLEG): container finished" podID="bf15ddf5-c075-4c71-9ab2-5e01a9637021" containerID="b3e56d858add0f47a4969696f7bff594aa4572fb548f309c4752e755481acf29" exitCode=0 Nov 02 22:56:23.483286 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:23.483271 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" event={"ID":"bf15ddf5-c075-4c71-9ab2-5e01a9637021","Type":"ContainerDied","Data":"b3e56d858add0f47a4969696f7bff594aa4572fb548f309c4752e755481acf29"} Nov 02 22:56:23.486898 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:23.486873 5672 generic.go:356] "Generic (PLEG): container finished" podID="510980f2-e363-4b29-8ee2-50f66965eb41" containerID="53fcf7604c9e44b7051c2698787c12d0599c70c06b845d949a4d04afa11569a6" exitCode=0 Nov 02 22:56:23.487038 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:23.486980 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" event={"ID":"510980f2-e363-4b29-8ee2-50f66965eb41","Type":"ContainerDied","Data":"53fcf7604c9e44b7051c2698787c12d0599c70c06b845d949a4d04afa11569a6"} Nov 02 22:56:23.489848 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:23.489829 5672 generic.go:356] "Generic (PLEG): container finished" podID="a63f6bbe-df96-4b1d-9da9-043ed361ca1d" containerID="330d17a45ef0b197795a783f3f449cc7e84576a452f040581fcc9d6d1d5164f4" exitCode=0 Nov 02 22:56:23.489925 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:23.489889 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" event={"ID":"a63f6bbe-df96-4b1d-9da9-043ed361ca1d","Type":"ContainerDied","Data":"330d17a45ef0b197795a783f3f449cc7e84576a452f040581fcc9d6d1d5164f4"} Nov 02 22:56:24.498430 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:24.498397 5672 generic.go:356] "Generic (PLEG): container finished" podID="e5c59539-2cdd-434b-ba77-e3585b8df55c" containerID="8c23422c197214d4f7977fe7b740f26b352c5e68c51982c3a52e31c3489846ce" exitCode=0 Nov 02 22:56:24.498797 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:24.498564 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" event={"ID":"e5c59539-2cdd-434b-ba77-e3585b8df55c","Type":"ContainerDied","Data":"8c23422c197214d4f7977fe7b740f26b352c5e68c51982c3a52e31c3489846ce"} Nov 02 22:56:24.502009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:24.501982 5672 generic.go:356] "Generic (PLEG): container finished" podID="820f8bff-cae1-448c-8c26-2b40775cba71" containerID="970c7ae5e042170a10267ba52706406ca6db6fb47fe7dcac2f54567ab34d05f6" exitCode=0 Nov 02 22:56:24.502111 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:24.502062 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" event={"ID":"820f8bff-cae1-448c-8c26-2b40775cba71","Type":"ContainerDied","Data":"970c7ae5e042170a10267ba52706406ca6db6fb47fe7dcac2f54567ab34d05f6"} Nov 02 22:56:24.504246 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:24.504223 5672 generic.go:356] "Generic (PLEG): container finished" podID="bee85a62-333f-4a03-8ef4-2908ebf1e990" containerID="f7f4b45d8be2e1ba22ac1b13e7e6d46b46f0da054315d2f194800f0503f38689" exitCode=0 Nov 02 22:56:24.504329 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:24.504258 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" event={"ID":"bee85a62-333f-4a03-8ef4-2908ebf1e990","Type":"ContainerDied","Data":"f7f4b45d8be2e1ba22ac1b13e7e6d46b46f0da054315d2f194800f0503f38689"} Nov 02 22:56:25.511746 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:25.511699 5672 generic.go:356] "Generic (PLEG): container finished" podID="bf15ddf5-c075-4c71-9ab2-5e01a9637021" containerID="9a1961e2181f41bf9a3006ef51e5dabecfa3e5e0e846dbc9f10aa5b591f0c3e4" exitCode=0 Nov 02 22:56:25.512226 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:25.511835 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" event={"ID":"bf15ddf5-c075-4c71-9ab2-5e01a9637021","Type":"ContainerDied","Data":"9a1961e2181f41bf9a3006ef51e5dabecfa3e5e0e846dbc9f10aa5b591f0c3e4"} Nov 02 22:56:25.514932 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:25.514908 5672 generic.go:356] "Generic (PLEG): container finished" podID="ef4c2feb-354b-449e-85ef-c503c66603f8" containerID="d0d182c015a61939a2b08e87d7114e6f497f08222c17aee70907ba623864bd71" exitCode=0 Nov 02 22:56:25.515017 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:25.514972 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" event={"ID":"ef4c2feb-354b-449e-85ef-c503c66603f8","Type":"ContainerDied","Data":"d0d182c015a61939a2b08e87d7114e6f497f08222c17aee70907ba623864bd71"} Nov 02 22:56:26.038968 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:56:26.038904 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:56:37.614180 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:37.614039 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" event={"ID":"510980f2-e363-4b29-8ee2-50f66965eb41","Type":"ContainerStarted","Data":"774b9ae7e1d1ab171f2ad091706cdd6a393f5e5dce0d06dd2028e8b6d69af7d6"} Nov 02 22:56:37.623769 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:37.623729 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" event={"ID":"a63f6bbe-df96-4b1d-9da9-043ed361ca1d","Type":"ContainerStarted","Data":"b5258f799fa95e687d66d245011e2941471ebc6f185655afe3df269484b55a47"} Nov 02 22:56:37.731289 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:37.731227 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" podStartSLOduration=2.366222539 podStartE2EDuration="32.731207813s" totalImagesPullingTime="30.364985274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:37.729020974 +0000 UTC m=+191.136332803" watchObservedRunningTime="2026-11-02 22:56:37.731207813 +0000 UTC m=+191.138519641" Nov 02 22:56:37.837653 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:37.836601 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" podStartSLOduration=2.721209998 podStartE2EDuration="32.836587307s" totalImagesPullingTime="30.115377309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:37.83633781 +0000 UTC m=+191.243649643" watchObservedRunningTime="2026-11-02 22:56:37.836587307 +0000 UTC m=+191.243899136" Nov 02 22:56:38.633201 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:38.632861 5672 generic.go:356] "Generic (PLEG): container finished" podID="510980f2-e363-4b29-8ee2-50f66965eb41" containerID="774b9ae7e1d1ab171f2ad091706cdd6a393f5e5dce0d06dd2028e8b6d69af7d6" exitCode=0 Nov 02 22:56:38.633201 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:38.632997 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" event={"ID":"510980f2-e363-4b29-8ee2-50f66965eb41","Type":"ContainerDied","Data":"774b9ae7e1d1ab171f2ad091706cdd6a393f5e5dce0d06dd2028e8b6d69af7d6"} Nov 02 22:56:38.636496 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:38.636466 5672 generic.go:356] "Generic (PLEG): container finished" podID="a63f6bbe-df96-4b1d-9da9-043ed361ca1d" containerID="b5258f799fa95e687d66d245011e2941471ebc6f185655afe3df269484b55a47" exitCode=0 Nov 02 22:56:38.636636 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:38.636507 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" event={"ID":"a63f6bbe-df96-4b1d-9da9-043ed361ca1d","Type":"ContainerDied","Data":"b5258f799fa95e687d66d245011e2941471ebc6f185655afe3df269484b55a47"} Nov 02 22:56:39.644261 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.644182 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" event={"ID":"820f8bff-cae1-448c-8c26-2b40775cba71","Type":"ContainerStarted","Data":"b7aee8ba16fac9077888119c80c4efba42412800ab088353ed70d7645ab6fed9"} Nov 02 22:56:39.835370 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.835343 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:39.850862 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.850841 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:39.915010 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.914941 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-util\" (UniqueName: \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-util\") pod \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " Nov 02 22:56:39.915010 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.914980 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510980f2-e363-4b29-8ee2-50f66965eb41-kube-api-access-786xf\" (UniqueName: \"kubernetes.io/projected/510980f2-e363-4b29-8ee2-50f66965eb41-kube-api-access-786xf\") pod \"510980f2-e363-4b29-8ee2-50f66965eb41\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " Nov 02 22:56:39.915197 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.915021 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-kube-api-access-bp8dd\" (UniqueName: \"kubernetes.io/projected/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-kube-api-access-bp8dd\") pod \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " Nov 02 22:56:39.915197 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.915066 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-util\" (UniqueName: \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-util\") pod \"510980f2-e363-4b29-8ee2-50f66965eb41\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " Nov 02 22:56:39.915197 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.915126 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-bundle\" (UniqueName: \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-bundle\") pod \"510980f2-e363-4b29-8ee2-50f66965eb41\" (UID: \"510980f2-e363-4b29-8ee2-50f66965eb41\") " Nov 02 22:56:39.915197 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.915165 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-bundle\") pod \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\" (UID: \"a63f6bbe-df96-4b1d-9da9-043ed361ca1d\") " Nov 02 22:56:39.915888 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.915853 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-bundle" pod "510980f2-e363-4b29-8ee2-50f66965eb41" (UID: "510980f2-e363-4b29-8ee2-50f66965eb41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:39.916333 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.916310 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-bundle" pod "a63f6bbe-df96-4b1d-9da9-043ed361ca1d" (UID: "a63f6bbe-df96-4b1d-9da9-043ed361ca1d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:39.921072 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.921050 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-util" pod "510980f2-e363-4b29-8ee2-50f66965eb41" (UID: "510980f2-e363-4b29-8ee2-50f66965eb41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:39.921514 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.921482 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510980f2-e363-4b29-8ee2-50f66965eb41-kube-api-access-786xf" pod "510980f2-e363-4b29-8ee2-50f66965eb41" (UID: "510980f2-e363-4b29-8ee2-50f66965eb41"). InnerVolumeSpecName "kube-api-access-786xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:39.921622 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.921603 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-util" pod "a63f6bbe-df96-4b1d-9da9-043ed361ca1d" (UID: "a63f6bbe-df96-4b1d-9da9-043ed361ca1d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:39.922334 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.922299 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-kube-api-access-bp8dd" pod "a63f6bbe-df96-4b1d-9da9-043ed361ca1d" (UID: "a63f6bbe-df96-4b1d-9da9-043ed361ca1d"). InnerVolumeSpecName "kube-api-access-bp8dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:39.942296 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:39.942230 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" podStartSLOduration=3.679445921 podStartE2EDuration="31.942199304s" totalImagesPullingTime="28.262753383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:39.753606226 +0000 UTC m=+193.160918055" watchObservedRunningTime="2026-11-02 22:56:39.942199304 +0000 UTC m=+193.349511134" Nov 02 22:56:40.016531 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.016497 5672 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-bundle\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:40.016531 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.016529 5672 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-util\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:40.016763 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.016545 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-786xf\" (UniqueName: \"kubernetes.io/projected/510980f2-e363-4b29-8ee2-50f66965eb41-kube-api-access-786xf\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:40.016763 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.016560 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp8dd\" (UniqueName: \"kubernetes.io/projected/a63f6bbe-df96-4b1d-9da9-043ed361ca1d-kube-api-access-bp8dd\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:40.016763 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.016575 5672 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-util\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:40.016763 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.016588 5672 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/510980f2-e363-4b29-8ee2-50f66965eb41-bundle\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:40.650439 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.650405 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" event={"ID":"a63f6bbe-df96-4b1d-9da9-043ed361ca1d","Type":"ContainerDied","Data":"06f83c14778f4deab4718e0a0f2df03dd5afc0b739e76b498dbb33b59088c94a"} Nov 02 22:56:40.650804 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.650458 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06f83c14778f4deab4718e0a0f2df03dd5afc0b739e76b498dbb33b59088c94a" Nov 02 22:56:40.650804 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.650420 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs85rd" Nov 02 22:56:40.652767 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.652744 5672 generic.go:356] "Generic (PLEG): container finished" podID="820f8bff-cae1-448c-8c26-2b40775cba71" containerID="b7aee8ba16fac9077888119c80c4efba42412800ab088353ed70d7645ab6fed9" exitCode=0 Nov 02 22:56:40.652882 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.652820 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" event={"ID":"820f8bff-cae1-448c-8c26-2b40775cba71","Type":"ContainerDied","Data":"b7aee8ba16fac9077888119c80c4efba42412800ab088353ed70d7645ab6fed9"} Nov 02 22:56:40.655899 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.655875 5672 generic.go:356] "Generic (PLEG): container finished" podID="bee85a62-333f-4a03-8ef4-2908ebf1e990" containerID="005c29fb6d3407b341bb967ca5a4068142cc8e7fb38a1e2ecc2bd20120f227a1" exitCode=0 Nov 02 22:56:40.656008 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.655946 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" event={"ID":"bee85a62-333f-4a03-8ef4-2908ebf1e990","Type":"ContainerDied","Data":"005c29fb6d3407b341bb967ca5a4068142cc8e7fb38a1e2ecc2bd20120f227a1"} Nov 02 22:56:40.658020 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.658000 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" event={"ID":"bf15ddf5-c075-4c71-9ab2-5e01a9637021","Type":"ContainerStarted","Data":"ad8df5470921438692f2666f3af1f15b9d4c4aa4599b8456b0cc15d9c3926ef6"} Nov 02 22:56:40.659882 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.659860 5672 generic.go:356] "Generic (PLEG): container finished" podID="e5c59539-2cdd-434b-ba77-e3585b8df55c" containerID="95b3e196635f6caa466c7ca1dc05bae1a74c301b887f64030eac2e6d724da198" exitCode=0 Nov 02 22:56:40.659974 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.659942 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" event={"ID":"e5c59539-2cdd-434b-ba77-e3585b8df55c","Type":"ContainerDied","Data":"95b3e196635f6caa466c7ca1dc05bae1a74c301b887f64030eac2e6d724da198"} Nov 02 22:56:40.661701 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.661684 5672 generic.go:356] "Generic (PLEG): container finished" podID="ef4c2feb-354b-449e-85ef-c503c66603f8" containerID="bd29778fa76b246285340b8327fa14cb1c354d3e99304669e1c9acfdae7d5fff" exitCode=0 Nov 02 22:56:40.661785 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.661741 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" event={"ID":"ef4c2feb-354b-449e-85ef-c503c66603f8","Type":"ContainerDied","Data":"bd29778fa76b246285340b8327fa14cb1c354d3e99304669e1c9acfdae7d5fff"} Nov 02 22:56:40.663401 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.663384 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" event={"ID":"510980f2-e363-4b29-8ee2-50f66965eb41","Type":"ContainerDied","Data":"2042b676b3a8b65ee52e3c818338898c124fd6b58f66a59186484b6d051a498e"} Nov 02 22:56:40.663401 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.663396 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c6c8m" Nov 02 22:56:40.663516 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:40.663410 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2042b676b3a8b65ee52e3c818338898c124fd6b58f66a59186484b6d051a498e" Nov 02 22:56:41.237250 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.237201 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" podStartSLOduration=4.113972174 podStartE2EDuration="33.23718751s" totalImagesPullingTime="29.123215336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:41.23653955 +0000 UTC m=+194.643851378" watchObservedRunningTime="2026-11-02 22:56:41.23718751 +0000 UTC m=+194.644499338" Nov 02 22:56:41.669552 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.669461 5672 generic.go:356] "Generic (PLEG): container finished" podID="bf15ddf5-c075-4c71-9ab2-5e01a9637021" containerID="ad8df5470921438692f2666f3af1f15b9d4c4aa4599b8456b0cc15d9c3926ef6" exitCode=0 Nov 02 22:56:41.669954 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.669557 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" event={"ID":"bf15ddf5-c075-4c71-9ab2-5e01a9637021","Type":"ContainerDied","Data":"ad8df5470921438692f2666f3af1f15b9d4c4aa4599b8456b0cc15d9c3926ef6"} Nov 02 22:56:41.820629 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.820604 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:41.848128 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.848104 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:41.854163 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.854132 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:41.856752 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.856734 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:41.935747 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.935682 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-util\" (UniqueName: \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-util\") pod \"820f8bff-cae1-448c-8c26-2b40775cba71\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " Nov 02 22:56:41.935747 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.935710 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-bundle\" (UniqueName: \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-bundle\") pod \"820f8bff-cae1-448c-8c26-2b40775cba71\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " Nov 02 22:56:41.935923 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.935759 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-bundle\") pod \"e5c59539-2cdd-434b-ba77-e3585b8df55c\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " Nov 02 22:56:41.935923 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.935776 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-bundle\") pod \"bee85a62-333f-4a03-8ef4-2908ebf1e990\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " Nov 02 22:56:41.935923 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.935812 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee85a62-333f-4a03-8ef4-2908ebf1e990-kube-api-access-dwqzw\" (UniqueName: \"kubernetes.io/projected/bee85a62-333f-4a03-8ef4-2908ebf1e990-kube-api-access-dwqzw\") pod \"bee85a62-333f-4a03-8ef4-2908ebf1e990\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " Nov 02 22:56:41.935923 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.935850 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820f8bff-cae1-448c-8c26-2b40775cba71-kube-api-access-jp9w2\" (UniqueName: \"kubernetes.io/projected/820f8bff-cae1-448c-8c26-2b40775cba71-kube-api-access-jp9w2\") pod \"820f8bff-cae1-448c-8c26-2b40775cba71\" (UID: \"820f8bff-cae1-448c-8c26-2b40775cba71\") " Nov 02 22:56:41.935923 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.935876 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c59539-2cdd-434b-ba77-e3585b8df55c-kube-api-access-rhmkn\" (UniqueName: \"kubernetes.io/projected/e5c59539-2cdd-434b-ba77-e3585b8df55c-kube-api-access-rhmkn\") pod \"e5c59539-2cdd-434b-ba77-e3585b8df55c\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " Nov 02 22:56:41.935923 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.935906 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-util\" (UniqueName: \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-util\") pod \"e5c59539-2cdd-434b-ba77-e3585b8df55c\" (UID: \"e5c59539-2cdd-434b-ba77-e3585b8df55c\") " Nov 02 22:56:41.936234 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.935942 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-util\" (UniqueName: \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-util\") pod \"ef4c2feb-354b-449e-85ef-c503c66603f8\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " Nov 02 22:56:41.936234 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.935996 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-bundle\") pod \"ef4c2feb-354b-449e-85ef-c503c66603f8\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " Nov 02 22:56:41.936234 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.936028 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4c2feb-354b-449e-85ef-c503c66603f8-kube-api-access-n5j8w\" (UniqueName: \"kubernetes.io/projected/ef4c2feb-354b-449e-85ef-c503c66603f8-kube-api-access-n5j8w\") pod \"ef4c2feb-354b-449e-85ef-c503c66603f8\" (UID: \"ef4c2feb-354b-449e-85ef-c503c66603f8\") " Nov 02 22:56:41.936234 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.936118 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-util\" (UniqueName: \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-util\") pod \"bee85a62-333f-4a03-8ef4-2908ebf1e990\" (UID: \"bee85a62-333f-4a03-8ef4-2908ebf1e990\") " Nov 02 22:56:41.936234 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.936169 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-bundle" pod "820f8bff-cae1-448c-8c26-2b40775cba71" (UID: "820f8bff-cae1-448c-8c26-2b40775cba71"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:41.936485 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.936393 5672 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-bundle\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:41.937129 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.937013 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-bundle" pod "ef4c2feb-354b-449e-85ef-c503c66603f8" (UID: "ef4c2feb-354b-449e-85ef-c503c66603f8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:41.937129 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.937102 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-bundle" pod "bee85a62-333f-4a03-8ef4-2908ebf1e990" (UID: "bee85a62-333f-4a03-8ef4-2908ebf1e990"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:41.937697 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.937660 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-bundle" pod "e5c59539-2cdd-434b-ba77-e3585b8df55c" (UID: "e5c59539-2cdd-434b-ba77-e3585b8df55c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:41.938674 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.938648 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4c2feb-354b-449e-85ef-c503c66603f8-kube-api-access-n5j8w" pod "ef4c2feb-354b-449e-85ef-c503c66603f8" (UID: "ef4c2feb-354b-449e-85ef-c503c66603f8"). InnerVolumeSpecName "kube-api-access-n5j8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:41.938752 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.938718 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee85a62-333f-4a03-8ef4-2908ebf1e990-kube-api-access-dwqzw" pod "bee85a62-333f-4a03-8ef4-2908ebf1e990" (UID: "bee85a62-333f-4a03-8ef4-2908ebf1e990"). InnerVolumeSpecName "kube-api-access-dwqzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:41.938787 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.938751 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820f8bff-cae1-448c-8c26-2b40775cba71-kube-api-access-jp9w2" pod "820f8bff-cae1-448c-8c26-2b40775cba71" (UID: "820f8bff-cae1-448c-8c26-2b40775cba71"). InnerVolumeSpecName "kube-api-access-jp9w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:41.939125 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.939107 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c59539-2cdd-434b-ba77-e3585b8df55c-kube-api-access-rhmkn" pod "e5c59539-2cdd-434b-ba77-e3585b8df55c" (UID: "e5c59539-2cdd-434b-ba77-e3585b8df55c"). InnerVolumeSpecName "kube-api-access-rhmkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:41.944794 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.944767 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-util" pod "bee85a62-333f-4a03-8ef4-2908ebf1e990" (UID: "bee85a62-333f-4a03-8ef4-2908ebf1e990"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:41.945675 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.945649 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-util" pod "ef4c2feb-354b-449e-85ef-c503c66603f8" (UID: "ef4c2feb-354b-449e-85ef-c503c66603f8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:41.946195 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.946175 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-util" pod "820f8bff-cae1-448c-8c26-2b40775cba71" (UID: "820f8bff-cae1-448c-8c26-2b40775cba71"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:41.947369 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:41.947348 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-util" pod "e5c59539-2cdd-434b-ba77-e3585b8df55c" (UID: "e5c59539-2cdd-434b-ba77-e3585b8df55c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:42.036968 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.036849 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp9w2\" (UniqueName: \"kubernetes.io/projected/820f8bff-cae1-448c-8c26-2b40775cba71-kube-api-access-jp9w2\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.036968 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.036878 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhmkn\" (UniqueName: \"kubernetes.io/projected/e5c59539-2cdd-434b-ba77-e3585b8df55c-kube-api-access-rhmkn\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.036968 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.036891 5672 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-util\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.036968 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.036905 5672 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-util\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.036968 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.036917 5672 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef4c2feb-354b-449e-85ef-c503c66603f8-bundle\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.036968 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.036931 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5j8w\" (UniqueName: \"kubernetes.io/projected/ef4c2feb-354b-449e-85ef-c503c66603f8-kube-api-access-n5j8w\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.036968 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.036942 5672 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-util\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.036968 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.036953 5672 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/820f8bff-cae1-448c-8c26-2b40775cba71-util\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.036968 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.036963 5672 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e5c59539-2cdd-434b-ba77-e3585b8df55c-bundle\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.036968 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.036973 5672 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bee85a62-333f-4a03-8ef4-2908ebf1e990-bundle\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.037478 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.036981 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwqzw\" (UniqueName: \"kubernetes.io/projected/bee85a62-333f-4a03-8ef4-2908ebf1e990-kube-api-access-dwqzw\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.675179 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.675135 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" Nov 02 22:56:42.675591 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.675132 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zkfvt" event={"ID":"820f8bff-cae1-448c-8c26-2b40775cba71","Type":"ContainerDied","Data":"24efbd8f531cd4f6997b847138ea86fac0894a99c5935fdcaaf26d233e2f994f"} Nov 02 22:56:42.675591 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.675271 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24efbd8f531cd4f6997b847138ea86fac0894a99c5935fdcaaf26d233e2f994f" Nov 02 22:56:42.677327 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.677308 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" Nov 02 22:56:42.677327 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.677322 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agcr98" event={"ID":"bee85a62-333f-4a03-8ef4-2908ebf1e990","Type":"ContainerDied","Data":"4a8d58d83b167d4737634973102a2cd779dbbffb39b6aef7ab6821af410abee0"} Nov 02 22:56:42.677536 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.677352 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a8d58d83b167d4737634973102a2cd779dbbffb39b6aef7ab6821af410abee0" Nov 02 22:56:42.679352 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.679329 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" Nov 02 22:56:42.679505 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.679358 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ct6mxn" event={"ID":"e5c59539-2cdd-434b-ba77-e3585b8df55c","Type":"ContainerDied","Data":"e935d89ce2ee3a8c43d7e0c0448f4eb6079e4724d190ba23829d7ac1b240fcdb"} Nov 02 22:56:42.679505 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.679395 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e935d89ce2ee3a8c43d7e0c0448f4eb6079e4724d190ba23829d7ac1b240fcdb" Nov 02 22:56:42.681416 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.681392 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" Nov 02 22:56:42.681416 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.681409 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zfn7l" event={"ID":"ef4c2feb-354b-449e-85ef-c503c66603f8","Type":"ContainerDied","Data":"3d896fa5bcdd0a84c8af7b59895763ff45727163286779fab63217561385a0ba"} Nov 02 22:56:42.681550 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.681447 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d896fa5bcdd0a84c8af7b59895763ff45727163286779fab63217561385a0ba" Nov 02 22:56:42.791593 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.791569 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:42.845246 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.845219 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-util\" (UniqueName: \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-util\") pod \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " Nov 02 22:56:42.845374 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.845282 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf15ddf5-c075-4c71-9ab2-5e01a9637021-kube-api-access-4dqtw\" (UniqueName: \"kubernetes.io/projected/bf15ddf5-c075-4c71-9ab2-5e01a9637021-kube-api-access-4dqtw\") pod \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " Nov 02 22:56:42.845374 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.845345 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-bundle\") pod \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\" (UID: \"bf15ddf5-c075-4c71-9ab2-5e01a9637021\") " Nov 02 22:56:42.845882 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.845857 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-bundle" pod "bf15ddf5-c075-4c71-9ab2-5e01a9637021" (UID: "bf15ddf5-c075-4c71-9ab2-5e01a9637021"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:42.847271 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.847245 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf15ddf5-c075-4c71-9ab2-5e01a9637021-kube-api-access-4dqtw" pod "bf15ddf5-c075-4c71-9ab2-5e01a9637021" (UID: "bf15ddf5-c075-4c71-9ab2-5e01a9637021"). InnerVolumeSpecName "kube-api-access-4dqtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:42.850479 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.850460 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-util" pod "bf15ddf5-c075-4c71-9ab2-5e01a9637021" (UID: "bf15ddf5-c075-4c71-9ab2-5e01a9637021"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:42.946428 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.946367 5672 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-util\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.946428 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.946390 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dqtw\" (UniqueName: \"kubernetes.io/projected/bf15ddf5-c075-4c71-9ab2-5e01a9637021-kube-api-access-4dqtw\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:42.946428 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:42.946399 5672 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf15ddf5-c075-4c71-9ab2-5e01a9637021-bundle\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:43.687104 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:43.687076 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" Nov 02 22:56:43.687598 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:43.687079 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sqk4g" event={"ID":"bf15ddf5-c075-4c71-9ab2-5e01a9637021","Type":"ContainerDied","Data":"8c93d34715cced77988090d043d0c51c5a8a045ef0577a2f443a4238952a5af8"} Nov 02 22:56:43.687598 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:43.687198 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c93d34715cced77988090d043d0c51c5a8a045ef0577a2f443a4238952a5af8" Nov 02 22:56:46.253336 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253301 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb"] Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253564 5672 cpu_manager.go:414] "containerMap: removing container" podUID="820f8bff-cae1-448c-8c26-2b40775cba71" containerName="util" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253574 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820f8bff-cae1-448c-8c26-2b40775cba71" containerName="util" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253590 5672 cpu_manager.go:414] "containerMap: removing container" podUID="510980f2-e363-4b29-8ee2-50f66965eb41" containerName="util" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253595 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510980f2-e363-4b29-8ee2-50f66965eb41" containerName="util" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253604 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bee85a62-333f-4a03-8ef4-2908ebf1e990" containerName="util" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253608 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee85a62-333f-4a03-8ef4-2908ebf1e990" containerName="util" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253614 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bee85a62-333f-4a03-8ef4-2908ebf1e990" containerName="pull" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253618 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee85a62-333f-4a03-8ef4-2908ebf1e990" containerName="pull" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253625 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a63f6bbe-df96-4b1d-9da9-043ed361ca1d" containerName="pull" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253629 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63f6bbe-df96-4b1d-9da9-043ed361ca1d" containerName="pull" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253635 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e5c59539-2cdd-434b-ba77-e3585b8df55c" containerName="pull" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253639 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c59539-2cdd-434b-ba77-e3585b8df55c" containerName="pull" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253644 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bf15ddf5-c075-4c71-9ab2-5e01a9637021" containerName="util" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253649 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf15ddf5-c075-4c71-9ab2-5e01a9637021" containerName="util" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253656 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bf15ddf5-c075-4c71-9ab2-5e01a9637021" containerName="pull" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253659 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf15ddf5-c075-4c71-9ab2-5e01a9637021" containerName="pull" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253664 5672 cpu_manager.go:414] "containerMap: removing container" podUID="820f8bff-cae1-448c-8c26-2b40775cba71" containerName="extract" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253668 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820f8bff-cae1-448c-8c26-2b40775cba71" containerName="extract" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253673 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e5c59539-2cdd-434b-ba77-e3585b8df55c" containerName="util" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253677 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c59539-2cdd-434b-ba77-e3585b8df55c" containerName="util" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253685 5672 cpu_manager.go:414] "containerMap: removing container" podUID="510980f2-e363-4b29-8ee2-50f66965eb41" containerName="pull" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253690 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510980f2-e363-4b29-8ee2-50f66965eb41" containerName="pull" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253697 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e5c59539-2cdd-434b-ba77-e3585b8df55c" containerName="extract" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253701 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c59539-2cdd-434b-ba77-e3585b8df55c" containerName="extract" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253706 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bf15ddf5-c075-4c71-9ab2-5e01a9637021" containerName="extract" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253710 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf15ddf5-c075-4c71-9ab2-5e01a9637021" containerName="extract" Nov 02 22:56:46.253707 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253720 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a63f6bbe-df96-4b1d-9da9-043ed361ca1d" containerName="util" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253724 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63f6bbe-df96-4b1d-9da9-043ed361ca1d" containerName="util" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253729 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ef4c2feb-354b-449e-85ef-c503c66603f8" containerName="pull" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253734 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4c2feb-354b-449e-85ef-c503c66603f8" containerName="pull" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253739 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bee85a62-333f-4a03-8ef4-2908ebf1e990" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253743 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee85a62-333f-4a03-8ef4-2908ebf1e990" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253750 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a63f6bbe-df96-4b1d-9da9-043ed361ca1d" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253755 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63f6bbe-df96-4b1d-9da9-043ed361ca1d" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253760 5672 cpu_manager.go:414] "containerMap: removing container" podUID="820f8bff-cae1-448c-8c26-2b40775cba71" containerName="pull" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253764 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820f8bff-cae1-448c-8c26-2b40775cba71" containerName="pull" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253774 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ef4c2feb-354b-449e-85ef-c503c66603f8" containerName="util" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253778 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4c2feb-354b-449e-85ef-c503c66603f8" containerName="util" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253785 5672 cpu_manager.go:414] "containerMap: removing container" podUID="510980f2-e363-4b29-8ee2-50f66965eb41" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253789 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510980f2-e363-4b29-8ee2-50f66965eb41" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253794 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ef4c2feb-354b-449e-85ef-c503c66603f8" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253798 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4c2feb-354b-449e-85ef-c503c66603f8" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253848 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c59539-2cdd-434b-ba77-e3585b8df55c" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253856 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf15ddf5-c075-4c71-9ab2-5e01a9637021" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253862 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee85a62-333f-4a03-8ef4-2908ebf1e990" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253869 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63f6bbe-df96-4b1d-9da9-043ed361ca1d" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253875 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4c2feb-354b-449e-85ef-c503c66603f8" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253880 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="510980f2-e363-4b29-8ee2-50f66965eb41" containerName="extract" Nov 02 22:56:46.254476 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.253887 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="820f8bff-cae1-448c-8c26-2b40775cba71" containerName="extract" Nov 02 22:56:46.268941 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.268918 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb" Nov 02 22:56:46.312034 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.312009 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb"] Nov 02 22:56:46.356886 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.356857 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:56:46.356979 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.356896 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 22:56:46.357103 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.357088 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-qzzvl\"" Nov 02 22:56:46.372458 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.372436 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e350b60-5c8d-4db7-ae48-7958e4b1d948-tmp\") pod \"jobset-operator-674cd59dc5-5bnqb\" (UID: \"4e350b60-5c8d-4db7-ae48-7958e4b1d948\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb" Nov 02 22:56:46.372527 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.372512 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lg4z\" (UniqueName: \"kubernetes.io/projected/4e350b60-5c8d-4db7-ae48-7958e4b1d948-kube-api-access-4lg4z\") pod \"jobset-operator-674cd59dc5-5bnqb\" (UID: \"4e350b60-5c8d-4db7-ae48-7958e4b1d948\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb" Nov 02 22:56:46.473437 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.473409 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lg4z\" (UniqueName: \"kubernetes.io/projected/4e350b60-5c8d-4db7-ae48-7958e4b1d948-kube-api-access-4lg4z\") pod \"jobset-operator-674cd59dc5-5bnqb\" (UID: \"4e350b60-5c8d-4db7-ae48-7958e4b1d948\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb" Nov 02 22:56:46.473532 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.473458 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e350b60-5c8d-4db7-ae48-7958e4b1d948-tmp\") pod \"jobset-operator-674cd59dc5-5bnqb\" (UID: \"4e350b60-5c8d-4db7-ae48-7958e4b1d948\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb" Nov 02 22:56:46.473817 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.473797 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e350b60-5c8d-4db7-ae48-7958e4b1d948-tmp\") pod \"jobset-operator-674cd59dc5-5bnqb\" (UID: \"4e350b60-5c8d-4db7-ae48-7958e4b1d948\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb" Nov 02 22:56:46.527009 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.526957 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lg4z\" (UniqueName: \"kubernetes.io/projected/4e350b60-5c8d-4db7-ae48-7958e4b1d948-kube-api-access-4lg4z\") pod \"jobset-operator-674cd59dc5-5bnqb\" (UID: \"4e350b60-5c8d-4db7-ae48-7958e4b1d948\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb" Nov 02 22:56:46.610570 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.610537 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb" Nov 02 22:56:46.969314 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:46.969284 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb"] Nov 02 22:56:46.972438 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:56:46.972416 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e350b60_5c8d_4db7_ae48_7958e4b1d948.slice/crio-58f77f808e298815a622c52c94c417da2db7f8bab0d8e6e4cca286f1fef118d2 WatchSource:0}: Error finding container 58f77f808e298815a622c52c94c417da2db7f8bab0d8e6e4cca286f1fef118d2: Status 404 returned error can't find the container with id 58f77f808e298815a622c52c94c417da2db7f8bab0d8e6e4cca286f1fef118d2 Nov 02 22:56:47.703606 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:47.703571 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb" event={"ID":"4e350b60-5c8d-4db7-ae48-7958e4b1d948","Type":"ContainerStarted","Data":"58f77f808e298815a622c52c94c417da2db7f8bab0d8e6e4cca286f1fef118d2"} Nov 02 22:56:49.706994 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.706957 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p"] Nov 02 22:56:49.713272 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.713240 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.761763 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.761738 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 22:56:49.762714 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.762685 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 22:56:49.762859 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.762839 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 22:56:49.799499 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.799469 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b6f698b-d190-4954-bc8c-342963324221-webhook-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.799666 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.799513 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b6f698b-d190-4954-bc8c-342963324221-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.799666 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.799536 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b6f698b-d190-4954-bc8c-342963324221-tmp\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.799666 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.799568 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b6f698b-d190-4954-bc8c-342963324221-metrics-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.799666 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.799603 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x77kr\" (UniqueName: \"kubernetes.io/projected/3b6f698b-d190-4954-bc8c-342963324221-kube-api-access-x77kr\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.799893 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.799807 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b6f698b-d190-4954-bc8c-342963324221-manifests\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.808301 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.808277 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 22:56:49.808446 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.808317 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 22:56:49.808555 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.808457 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-lpc6c\"" Nov 02 22:56:49.848703 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.848673 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p"] Nov 02 22:56:49.900546 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.900517 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b6f698b-d190-4954-bc8c-342963324221-manifests\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.900546 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.900556 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b6f698b-d190-4954-bc8c-342963324221-webhook-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.900799 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.900583 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b6f698b-d190-4954-bc8c-342963324221-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.900799 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.900709 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b6f698b-d190-4954-bc8c-342963324221-tmp\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.900799 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.900777 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b6f698b-d190-4954-bc8c-342963324221-metrics-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.900959 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.900810 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x77kr\" (UniqueName: \"kubernetes.io/projected/3b6f698b-d190-4954-bc8c-342963324221-kube-api-access-x77kr\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.901018 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.900997 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3b6f698b-d190-4954-bc8c-342963324221-manifests\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.903205 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.903179 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b6f698b-d190-4954-bc8c-342963324221-tmp\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.903632 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.903606 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3b6f698b-d190-4954-bc8c-342963324221-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.903632 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.903618 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b6f698b-d190-4954-bc8c-342963324221-webhook-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.903802 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.903733 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b6f698b-d190-4954-bc8c-342963324221-metrics-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:49.957438 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:49.957375 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x77kr\" (UniqueName: \"kubernetes.io/projected/3b6f698b-d190-4954-bc8c-342963324221-kube-api-access-x77kr\") pod \"opendatahub-operator-controller-manager-7858c78b8d-bm69p\" (UID: \"3b6f698b-d190-4954-bc8c-342963324221\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:50.031506 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:50.031471 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:50.507011 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:50.506985 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p"] Nov 02 22:56:50.509720 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:56:50.509683 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6f698b_d190_4954_bc8c_342963324221.slice/crio-52fc25129f703ece3ec66c620a4fb800a3252244a8047259f12e806981dbe4aa WatchSource:0}: Error finding container 52fc25129f703ece3ec66c620a4fb800a3252244a8047259f12e806981dbe4aa: Status 404 returned error can't find the container with id 52fc25129f703ece3ec66c620a4fb800a3252244a8047259f12e806981dbe4aa Nov 02 22:56:50.721611 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:50.721575 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" event={"ID":"3b6f698b-d190-4954-bc8c-342963324221","Type":"ContainerStarted","Data":"52fc25129f703ece3ec66c620a4fb800a3252244a8047259f12e806981dbe4aa"} Nov 02 22:56:50.723232 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:50.723207 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb" event={"ID":"4e350b60-5c8d-4db7-ae48-7958e4b1d948","Type":"ContainerStarted","Data":"f387e535a16d2773290973361abc618662ac615d42d37bd3bc7e96841dfbb002"} Nov 02 22:56:50.839511 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:50.839457 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5bnqb" podStartSLOduration=1.492602956 podStartE2EDuration="4.839440065s" totalImagesPullingTime="3.346837109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:50.837562309 +0000 UTC m=+204.244874142" watchObservedRunningTime="2026-11-02 22:56:50.839440065 +0000 UTC m=+204.246751894" Nov 02 22:56:54.744303 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:54.744231 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" event={"ID":"3b6f698b-d190-4954-bc8c-342963324221","Type":"ContainerStarted","Data":"4473431e5afe004d0dc75115bd4a96ed77a6ed43ab3b57fccfbfe5436d01ec7f"} Nov 02 22:56:55.750087 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:55.750053 5672 generic.go:356] "Generic (PLEG): container finished" podID="3b6f698b-d190-4954-bc8c-342963324221" containerID="4473431e5afe004d0dc75115bd4a96ed77a6ed43ab3b57fccfbfe5436d01ec7f" exitCode=0 Nov 02 22:56:55.750512 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:55.750136 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" event={"ID":"3b6f698b-d190-4954-bc8c-342963324221","Type":"ContainerDied","Data":"4473431e5afe004d0dc75115bd4a96ed77a6ed43ab3b57fccfbfe5436d01ec7f"} Nov 02 22:56:56.019188 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:56:56.019085 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:56:56.757085 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:56.757059 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" event={"ID":"3b6f698b-d190-4954-bc8c-342963324221","Type":"ContainerStarted","Data":"9efde61896575eb3585006cab5d7ee0b3fd0097db218e452972e91d9f0d1fd4c"} Nov 02 22:56:56.757403 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:56.757288 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:56:56.980797 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:56:56.980745 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" podStartSLOduration=2.582566235 podStartE2EDuration="7.980731753s" totalImagesPullingTime="4.907425683s" totalInitContainerRuntime="490.739835ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:56.979782286 +0000 UTC m=+210.387094115" watchObservedRunningTime="2026-11-02 22:56:56.980731753 +0000 UTC m=+210.388043583" Nov 02 22:57:07.764096 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:07.764065 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-bm69p" Nov 02 22:57:09.534422 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.534388 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv"] Nov 02 22:57:09.538788 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.538770 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 02 22:57:09.583091 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.583063 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv"] Nov 02 22:57:09.628069 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.628038 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 22:57:09.628247 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.628191 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 22:57:09.628308 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.628260 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x7x5p\"" Nov 02 22:57:09.629470 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.629452 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c"] Nov 02 22:57:09.633383 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.633365 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" Nov 02 22:57:09.662183 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.662162 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c"] Nov 02 22:57:09.673158 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.673129 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/027227b3-6a47-4b4a-8506-c8bf8763f4d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rqrwv\" (UID: \"027227b3-6a47-4b4a-8506-c8bf8763f4d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 02 22:57:09.673256 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.673237 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcvkd\" (UniqueName: \"kubernetes.io/projected/027227b3-6a47-4b4a-8506-c8bf8763f4d3-kube-api-access-pcvkd\") pod \"kuadrant-operator-controller-manager-678d87766f-rqrwv\" (UID: \"027227b3-6a47-4b4a-8506-c8bf8763f4d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 02 22:57:09.690767 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.690747 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 22:57:09.690865 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.690792 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 22:57:09.690865 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.690804 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5f8dz\"" Nov 02 22:57:09.773960 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.773917 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/027227b3-6a47-4b4a-8506-c8bf8763f4d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rqrwv\" (UID: \"027227b3-6a47-4b4a-8506-c8bf8763f4d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 02 22:57:09.774075 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.773986 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcvkd\" (UniqueName: \"kubernetes.io/projected/027227b3-6a47-4b4a-8506-c8bf8763f4d3-kube-api-access-pcvkd\") pod \"kuadrant-operator-controller-manager-678d87766f-rqrwv\" (UID: \"027227b3-6a47-4b4a-8506-c8bf8763f4d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 02 22:57:09.774075 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.774021 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03c485fc-d4a2-4e23-8c7a-4db4f6b079bc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bgg4c\" (UID: \"03c485fc-d4a2-4e23-8c7a-4db4f6b079bc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" Nov 02 22:57:09.774254 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.774107 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdddx\" (UniqueName: \"kubernetes.io/projected/03c485fc-d4a2-4e23-8c7a-4db4f6b079bc-kube-api-access-jdddx\") pod \"cert-manager-webhook-5dbbbf75cc-bgg4c\" (UID: \"03c485fc-d4a2-4e23-8c7a-4db4f6b079bc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" Nov 02 22:57:09.774522 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.774500 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/027227b3-6a47-4b4a-8506-c8bf8763f4d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rqrwv\" (UID: \"027227b3-6a47-4b4a-8506-c8bf8763f4d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 02 22:57:09.828333 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.828270 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcvkd\" (UniqueName: \"kubernetes.io/projected/027227b3-6a47-4b4a-8506-c8bf8763f4d3-kube-api-access-pcvkd\") pod \"kuadrant-operator-controller-manager-678d87766f-rqrwv\" (UID: \"027227b3-6a47-4b4a-8506-c8bf8763f4d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 02 22:57:09.852068 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.852042 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 02 22:57:09.874593 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.874570 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdddx\" (UniqueName: \"kubernetes.io/projected/03c485fc-d4a2-4e23-8c7a-4db4f6b079bc-kube-api-access-jdddx\") pod \"cert-manager-webhook-5dbbbf75cc-bgg4c\" (UID: \"03c485fc-d4a2-4e23-8c7a-4db4f6b079bc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" Nov 02 22:57:09.874687 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.874629 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03c485fc-d4a2-4e23-8c7a-4db4f6b079bc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bgg4c\" (UID: \"03c485fc-d4a2-4e23-8c7a-4db4f6b079bc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" Nov 02 22:57:09.931559 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.931533 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdddx\" (UniqueName: \"kubernetes.io/projected/03c485fc-d4a2-4e23-8c7a-4db4f6b079bc-kube-api-access-jdddx\") pod \"cert-manager-webhook-5dbbbf75cc-bgg4c\" (UID: \"03c485fc-d4a2-4e23-8c7a-4db4f6b079bc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" Nov 02 22:57:09.935760 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.935731 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03c485fc-d4a2-4e23-8c7a-4db4f6b079bc-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bgg4c\" (UID: \"03c485fc-d4a2-4e23-8c7a-4db4f6b079bc\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" Nov 02 22:57:09.945042 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:09.944810 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" Nov 02 22:57:10.339603 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:57:10.339576 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027227b3_6a47_4b4a_8506_c8bf8763f4d3.slice/crio-69e676792526ea427b6b915b8e614d8646bd629b1ca202e7582c41112b78ecab WatchSource:0}: Error finding container 69e676792526ea427b6b915b8e614d8646bd629b1ca202e7582c41112b78ecab: Status 404 returned error can't find the container with id 69e676792526ea427b6b915b8e614d8646bd629b1ca202e7582c41112b78ecab Nov 02 22:57:10.342238 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:10.342211 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv"] Nov 02 22:57:10.373198 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:10.373175 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c"] Nov 02 22:57:10.385805 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:57:10.385779 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03c485fc_d4a2_4e23_8c7a_4db4f6b079bc.slice/crio-f43df05ea18f120762fbb3b2763a9a34de0e856221e00e9a22091fb4a5ef5893 WatchSource:0}: Error finding container f43df05ea18f120762fbb3b2763a9a34de0e856221e00e9a22091fb4a5ef5893: Status 404 returned error can't find the container with id f43df05ea18f120762fbb3b2763a9a34de0e856221e00e9a22091fb4a5ef5893 Nov 02 22:57:10.815766 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:10.815731 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" event={"ID":"03c485fc-d4a2-4e23-8c7a-4db4f6b079bc","Type":"ContainerStarted","Data":"f43df05ea18f120762fbb3b2763a9a34de0e856221e00e9a22091fb4a5ef5893"} Nov 02 22:57:10.817047 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:10.817022 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" event={"ID":"027227b3-6a47-4b4a-8506-c8bf8763f4d3","Type":"ContainerStarted","Data":"69e676792526ea427b6b915b8e614d8646bd629b1ca202e7582c41112b78ecab"} Nov 02 22:57:14.840971 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:14.840932 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" event={"ID":"03c485fc-d4a2-4e23-8c7a-4db4f6b079bc","Type":"ContainerStarted","Data":"ed5b4b0b4c6fd86b961fecaf714a51fbecee029f2a54bf87887720dea00d8d11"} Nov 02 22:57:14.841487 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:14.841068 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" Nov 02 22:57:14.948126 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:14.948071 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" podStartSLOduration=1.622215505 podStartE2EDuration="5.948057753s" totalImagesPullingTime="4.325842248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:14.946985618 +0000 UTC m=+228.354297454" watchObservedRunningTime="2026-11-02 22:57:14.948057753 +0000 UTC m=+228.355369582" Nov 02 22:57:18.863533 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:18.863485 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" event={"ID":"027227b3-6a47-4b4a-8506-c8bf8763f4d3","Type":"ContainerStarted","Data":"e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f"} Nov 02 22:57:18.863989 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:18.863608 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 02 22:57:19.067324 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:19.067267 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" podStartSLOduration=2.456207587 podStartE2EDuration="10.067254122s" totalImagesPullingTime="7.611046535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:19.019270586 +0000 UTC m=+232.426582418" watchObservedRunningTime="2026-11-02 22:57:19.067254122 +0000 UTC m=+232.474565949" Nov 02 22:57:20.848905 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:20.848873 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bgg4c" Nov 02 22:57:22.523628 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.523594 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z"] Nov 02 22:57:22.527396 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.527371 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" Nov 02 22:57:22.576063 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.576037 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 22:57:22.576063 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.576051 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 22:57:22.578924 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.578906 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-clk27\"" Nov 02 22:57:22.580798 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.580778 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z"] Nov 02 22:57:22.685821 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.685791 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c50862d-0486-4a19-9f75-4bf4f6d27dc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-kbd6z\" (UID: \"2c50862d-0486-4a19-9f75-4bf4f6d27dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" Nov 02 22:57:22.685965 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.685840 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxrr\" (UniqueName: \"kubernetes.io/projected/2c50862d-0486-4a19-9f75-4bf4f6d27dc2-kube-api-access-xhxrr\") pod \"kuadrant-console-plugin-66c4fb575d-kbd6z\" (UID: \"2c50862d-0486-4a19-9f75-4bf4f6d27dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" Nov 02 22:57:22.685965 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.685922 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c50862d-0486-4a19-9f75-4bf4f6d27dc2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-kbd6z\" (UID: \"2c50862d-0486-4a19-9f75-4bf4f6d27dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" Nov 02 22:57:22.786685 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.786605 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxrr\" (UniqueName: \"kubernetes.io/projected/2c50862d-0486-4a19-9f75-4bf4f6d27dc2-kube-api-access-xhxrr\") pod \"kuadrant-console-plugin-66c4fb575d-kbd6z\" (UID: \"2c50862d-0486-4a19-9f75-4bf4f6d27dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" Nov 02 22:57:22.786813 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.786684 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c50862d-0486-4a19-9f75-4bf4f6d27dc2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-kbd6z\" (UID: \"2c50862d-0486-4a19-9f75-4bf4f6d27dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" Nov 02 22:57:22.786813 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.786736 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c50862d-0486-4a19-9f75-4bf4f6d27dc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-kbd6z\" (UID: \"2c50862d-0486-4a19-9f75-4bf4f6d27dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" Nov 02 22:57:22.787354 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.787331 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c50862d-0486-4a19-9f75-4bf4f6d27dc2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-kbd6z\" (UID: \"2c50862d-0486-4a19-9f75-4bf4f6d27dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" Nov 02 22:57:22.789075 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.789053 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c50862d-0486-4a19-9f75-4bf4f6d27dc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-kbd6z\" (UID: \"2c50862d-0486-4a19-9f75-4bf4f6d27dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" Nov 02 22:57:22.839133 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:22.839108 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxrr\" (UniqueName: \"kubernetes.io/projected/2c50862d-0486-4a19-9f75-4bf4f6d27dc2-kube-api-access-xhxrr\") pod \"kuadrant-console-plugin-66c4fb575d-kbd6z\" (UID: \"2c50862d-0486-4a19-9f75-4bf4f6d27dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" Nov 02 22:57:23.138664 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:23.138588 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" Nov 02 22:57:23.505234 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:23.505199 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z"] Nov 02 22:57:23.507988 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:57:23.507961 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c50862d_0486_4a19_9f75_4bf4f6d27dc2.slice/crio-fbaa6be0a58cc2008e7b74a01a3caa2dd6d8e90549fba3bf6e1d22c8abca352d WatchSource:0}: Error finding container fbaa6be0a58cc2008e7b74a01a3caa2dd6d8e90549fba3bf6e1d22c8abca352d: Status 404 returned error can't find the container with id fbaa6be0a58cc2008e7b74a01a3caa2dd6d8e90549fba3bf6e1d22c8abca352d Nov 02 22:57:23.885363 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:23.885285 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" event={"ID":"2c50862d-0486-4a19-9f75-4bf4f6d27dc2","Type":"ContainerStarted","Data":"fbaa6be0a58cc2008e7b74a01a3caa2dd6d8e90549fba3bf6e1d22c8abca352d"} Nov 02 22:57:26.031016 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:57:26.030945 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:57:26.995984 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:26.995950 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls"] Nov 02 22:57:26.999288 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:26.999264 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" Nov 02 22:57:27.076294 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.076252 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls"] Nov 02 22:57:27.119307 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.119265 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 22:57:27.119569 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.119549 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jx5r7\"" Nov 02 22:57:27.125271 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.125245 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d22427e6-5f74-40a2-a89b-68bb7d372b0d-tls-certs\") pod \"kube-auth-proxy-7cc54b748b-tzzls\" (UID: \"d22427e6-5f74-40a2-a89b-68bb7d372b0d\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" Nov 02 22:57:27.125412 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.125276 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p922p\" (UniqueName: \"kubernetes.io/projected/d22427e6-5f74-40a2-a89b-68bb7d372b0d-kube-api-access-p922p\") pod \"kube-auth-proxy-7cc54b748b-tzzls\" (UID: \"d22427e6-5f74-40a2-a89b-68bb7d372b0d\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" Nov 02 22:57:27.125412 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.125323 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d22427e6-5f74-40a2-a89b-68bb7d372b0d-tmp\") pod \"kube-auth-proxy-7cc54b748b-tzzls\" (UID: \"d22427e6-5f74-40a2-a89b-68bb7d372b0d\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" Nov 02 22:57:27.150450 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.150421 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 22:57:27.226915 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.226880 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d22427e6-5f74-40a2-a89b-68bb7d372b0d-tls-certs\") pod \"kube-auth-proxy-7cc54b748b-tzzls\" (UID: \"d22427e6-5f74-40a2-a89b-68bb7d372b0d\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" Nov 02 22:57:27.227086 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.226935 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p922p\" (UniqueName: \"kubernetes.io/projected/d22427e6-5f74-40a2-a89b-68bb7d372b0d-kube-api-access-p922p\") pod \"kube-auth-proxy-7cc54b748b-tzzls\" (UID: \"d22427e6-5f74-40a2-a89b-68bb7d372b0d\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" Nov 02 22:57:27.227086 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.227008 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d22427e6-5f74-40a2-a89b-68bb7d372b0d-tmp\") pod \"kube-auth-proxy-7cc54b748b-tzzls\" (UID: \"d22427e6-5f74-40a2-a89b-68bb7d372b0d\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" Nov 02 22:57:27.229295 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.229258 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d22427e6-5f74-40a2-a89b-68bb7d372b0d-tmp\") pod \"kube-auth-proxy-7cc54b748b-tzzls\" (UID: \"d22427e6-5f74-40a2-a89b-68bb7d372b0d\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" Nov 02 22:57:27.229495 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.229476 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d22427e6-5f74-40a2-a89b-68bb7d372b0d-tls-certs\") pod \"kube-auth-proxy-7cc54b748b-tzzls\" (UID: \"d22427e6-5f74-40a2-a89b-68bb7d372b0d\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" Nov 02 22:57:27.300302 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.300229 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p922p\" (UniqueName: \"kubernetes.io/projected/d22427e6-5f74-40a2-a89b-68bb7d372b0d-kube-api-access-p922p\") pod \"kube-auth-proxy-7cc54b748b-tzzls\" (UID: \"d22427e6-5f74-40a2-a89b-68bb7d372b0d\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" Nov 02 22:57:27.312858 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.312834 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" Nov 02 22:57:27.705574 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.705502 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls"] Nov 02 22:57:27.715972 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:57:27.715934 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22427e6_5f74_40a2_a89b_68bb7d372b0d.slice/crio-08caac3ad104e825b33d0b1fe528176faf306314d18050b8d1b2afda952451b6 WatchSource:0}: Error finding container 08caac3ad104e825b33d0b1fe528176faf306314d18050b8d1b2afda952451b6: Status 404 returned error can't find the container with id 08caac3ad104e825b33d0b1fe528176faf306314d18050b8d1b2afda952451b6 Nov 02 22:57:27.914698 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:27.914655 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" event={"ID":"d22427e6-5f74-40a2-a89b-68bb7d372b0d","Type":"ContainerStarted","Data":"08caac3ad104e825b33d0b1fe528176faf306314d18050b8d1b2afda952451b6"} Nov 02 22:57:29.870057 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:29.870035 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 02 22:57:29.926819 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:29.926784 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" event={"ID":"2c50862d-0486-4a19-9f75-4bf4f6d27dc2","Type":"ContainerStarted","Data":"03a35e6cc21283f49a9d0290825bb2493925a4d9da4cd4a801f2f8230616f0a1"} Nov 02 22:57:30.158865 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:30.158749 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kbd6z" podStartSLOduration=1.862151524 podStartE2EDuration="8.158728955s" totalImagesPullingTime="6.296577431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:30.157653244 +0000 UTC m=+243.564965075" watchObservedRunningTime="2026-11-02 22:57:30.158728955 +0000 UTC m=+243.566040835" Nov 02 22:57:31.937842 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:31.937803 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" event={"ID":"d22427e6-5f74-40a2-a89b-68bb7d372b0d","Type":"ContainerStarted","Data":"6966125918d02ec9629d7af28d1c2654c1a78f5954b272d9c9561f19cb5a52c6"} Nov 02 22:57:32.053870 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:32.053822 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cc54b748b-tzzls" podStartSLOduration=2.654797688 podStartE2EDuration="6.053808627s" totalImagesPullingTime="3.399010939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:32.052944543 +0000 UTC m=+245.460256373" watchObservedRunningTime="2026-11-02 22:57:32.053808627 +0000 UTC m=+245.461120457" Nov 02 22:57:46.003265 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.003233 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 22:57:46.051506 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.051479 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" Nov 02 22:57:46.060663 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.060637 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 22:57:46.104792 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.104767 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 22:57:46.157038 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.157011 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e9db611-6d4f-4748-a3f4-4e8cfff61634-config-file\") pod \"limitador-limitador-7c9b87d574-6b47m\" (UID: \"8e9db611-6d4f-4748-a3f4-4e8cfff61634\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" Nov 02 22:57:46.157219 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.157109 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfx9m\" (UniqueName: \"kubernetes.io/projected/8e9db611-6d4f-4748-a3f4-4e8cfff61634-kube-api-access-bfx9m\") pod \"limitador-limitador-7c9b87d574-6b47m\" (UID: \"8e9db611-6d4f-4748-a3f4-4e8cfff61634\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" Nov 02 22:57:46.258177 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.258099 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e9db611-6d4f-4748-a3f4-4e8cfff61634-config-file\") pod \"limitador-limitador-7c9b87d574-6b47m\" (UID: \"8e9db611-6d4f-4748-a3f4-4e8cfff61634\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" Nov 02 22:57:46.258301 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.258250 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfx9m\" (UniqueName: \"kubernetes.io/projected/8e9db611-6d4f-4748-a3f4-4e8cfff61634-kube-api-access-bfx9m\") pod \"limitador-limitador-7c9b87d574-6b47m\" (UID: \"8e9db611-6d4f-4748-a3f4-4e8cfff61634\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" Nov 02 22:57:46.258871 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.258846 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e9db611-6d4f-4748-a3f4-4e8cfff61634-config-file\") pod \"limitador-limitador-7c9b87d574-6b47m\" (UID: \"8e9db611-6d4f-4748-a3f4-4e8cfff61634\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" Nov 02 22:57:46.304140 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.304110 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 22:57:46.311632 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.311612 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfx9m\" (UniqueName: \"kubernetes.io/projected/8e9db611-6d4f-4748-a3f4-4e8cfff61634-kube-api-access-bfx9m\") pod \"limitador-limitador-7c9b87d574-6b47m\" (UID: \"8e9db611-6d4f-4748-a3f4-4e8cfff61634\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" Nov 02 22:57:46.378271 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.378249 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" Nov 02 22:57:46.738420 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:46.738229 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 22:57:46.740675 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:57:46.740644 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9db611_6d4f_4748_a3f4_4e8cfff61634.slice/crio-e3ed632c67811de2fa7caa345fbba1d40d8ddeba3e1a57b33400d51fa583d04b WatchSource:0}: Error finding container e3ed632c67811de2fa7caa345fbba1d40d8ddeba3e1a57b33400d51fa583d04b: Status 404 returned error can't find the container with id e3ed632c67811de2fa7caa345fbba1d40d8ddeba3e1a57b33400d51fa583d04b Nov 02 22:57:47.029111 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:47.029029 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" event={"ID":"8e9db611-6d4f-4748-a3f4-4e8cfff61634","Type":"ContainerStarted","Data":"e3ed632c67811de2fa7caa345fbba1d40d8ddeba3e1a57b33400d51fa583d04b"} Nov 02 22:57:50.045926 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:50.045887 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" event={"ID":"8e9db611-6d4f-4748-a3f4-4e8cfff61634","Type":"ContainerStarted","Data":"70f7af70c9b82c2e40b22496f742cdb1f27c5f5fc2bbdfbc1df48e1973e5740c"} Nov 02 22:57:50.046420 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:50.045966 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" Nov 02 22:57:50.154066 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:57:50.154006 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" podStartSLOduration=2.7612268909999997 podStartE2EDuration="5.153989206s" totalImagesPullingTime="2.392762315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:50.152856515 +0000 UTC m=+263.560168345" watchObservedRunningTime="2026-11-02 22:57:50.153989206 +0000 UTC m=+263.561301083" Nov 02 22:57:56.018801 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:57:56.018761 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:58:01.051727 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:01.051694 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6b47m" Nov 02 22:58:26.019746 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:58:26.019670 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:58:29.603082 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.603052 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66c9557c77-n49l7"] Nov 02 22:58:29.609207 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.609187 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:29.629841 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.629813 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxpss\" (UniqueName: \"kubernetes.io/projected/f3011c35-3232-4fd7-81be-b178a0dad4d4-kube-api-access-xxpss\") pod \"maas-controller-66c9557c77-n49l7\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:29.629952 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.629890 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-metrics-certs\") pod \"maas-controller-66c9557c77-n49l7\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:29.629952 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.629945 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-webhook-cert\") pod \"maas-controller-66c9557c77-n49l7\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:29.658421 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.658392 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 22:58:29.658421 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.658400 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8fxqx\"" Nov 02 22:58:29.658557 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.658492 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 22:58:29.704815 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.704785 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 22:58:29.704943 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.704879 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 22:58:29.704943 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.704879 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 22:58:29.706419 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.706399 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66c9557c77-n49l7"] Nov 02 22:58:29.730533 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.730511 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxpss\" (UniqueName: \"kubernetes.io/projected/f3011c35-3232-4fd7-81be-b178a0dad4d4-kube-api-access-xxpss\") pod \"maas-controller-66c9557c77-n49l7\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:29.730602 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.730576 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-metrics-certs\") pod \"maas-controller-66c9557c77-n49l7\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:29.730602 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.730596 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-webhook-cert\") pod \"maas-controller-66c9557c77-n49l7\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:29.733061 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.733032 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-metrics-certs\") pod \"maas-controller-66c9557c77-n49l7\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:29.733182 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.733064 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-webhook-cert\") pod \"maas-controller-66c9557c77-n49l7\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:29.783472 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.783441 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxpss\" (UniqueName: \"kubernetes.io/projected/f3011c35-3232-4fd7-81be-b178a0dad4d4-kube-api-access-xxpss\") pod \"maas-controller-66c9557c77-n49l7\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:29.921714 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:29.921629 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:30.180542 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.180472 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b67978596-nrwt6"] Nov 02 22:58:30.185424 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.185402 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:30.235419 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.235394 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qp6\" (UniqueName: \"kubernetes.io/projected/906286bb-6770-4daa-aa76-514a51f1347e-kube-api-access-d5qp6\") pod \"maas-controller-b67978596-nrwt6\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:30.235546 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.235440 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-webhook-cert\") pod \"maas-controller-b67978596-nrwt6\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:30.235583 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.235538 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-metrics-certs\") pod \"maas-controller-b67978596-nrwt6\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:30.238405 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.238383 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b67978596-nrwt6"] Nov 02 22:58:30.336515 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.336485 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-webhook-cert\") pod \"maas-controller-b67978596-nrwt6\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:30.336719 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.336577 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-metrics-certs\") pod \"maas-controller-b67978596-nrwt6\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:30.336719 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.336613 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qp6\" (UniqueName: \"kubernetes.io/projected/906286bb-6770-4daa-aa76-514a51f1347e-kube-api-access-d5qp6\") pod \"maas-controller-b67978596-nrwt6\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:30.337001 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.336972 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66c9557c77-n49l7"] Nov 02 22:58:30.339665 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.339640 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-webhook-cert\") pod \"maas-controller-b67978596-nrwt6\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:30.339786 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.339649 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-metrics-certs\") pod \"maas-controller-b67978596-nrwt6\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:30.340326 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:58:30.340303 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3011c35_3232_4fd7_81be_b178a0dad4d4.slice/crio-1bc7d3b37788383cf0441f81fa646f8cab62e0145dce51a3d97fcc2b3dc90e36 WatchSource:0}: Error finding container 1bc7d3b37788383cf0441f81fa646f8cab62e0145dce51a3d97fcc2b3dc90e36: Status 404 returned error can't find the container with id 1bc7d3b37788383cf0441f81fa646f8cab62e0145dce51a3d97fcc2b3dc90e36 Nov 02 22:58:30.389345 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.389319 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qp6\" (UniqueName: \"kubernetes.io/projected/906286bb-6770-4daa-aa76-514a51f1347e-kube-api-access-d5qp6\") pod \"maas-controller-b67978596-nrwt6\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:30.498117 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.498086 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:30.859430 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:30.859371 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b67978596-nrwt6"] Nov 02 22:58:30.861623 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:58:30.861597 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906286bb_6770_4daa_aa76_514a51f1347e.slice/crio-d0f21fb3d134dfd8092452ee1b94e370bfcc4e30ffa46443e1b393784f677fc8 WatchSource:0}: Error finding container d0f21fb3d134dfd8092452ee1b94e370bfcc4e30ffa46443e1b393784f677fc8: Status 404 returned error can't find the container with id d0f21fb3d134dfd8092452ee1b94e370bfcc4e30ffa46443e1b393784f677fc8 Nov 02 22:58:31.231959 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:31.231920 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b67978596-nrwt6" event={"ID":"906286bb-6770-4daa-aa76-514a51f1347e","Type":"ContainerStarted","Data":"d0f21fb3d134dfd8092452ee1b94e370bfcc4e30ffa46443e1b393784f677fc8"} Nov 02 22:58:31.233114 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:31.233090 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c9557c77-n49l7" event={"ID":"f3011c35-3232-4fd7-81be-b178a0dad4d4","Type":"ContainerStarted","Data":"1bc7d3b37788383cf0441f81fa646f8cab62e0145dce51a3d97fcc2b3dc90e36"} Nov 02 22:58:37.267632 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:37.267598 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b67978596-nrwt6" event={"ID":"906286bb-6770-4daa-aa76-514a51f1347e","Type":"ContainerStarted","Data":"737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d"} Nov 02 22:58:37.268076 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:37.267716 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:37.269311 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:37.269289 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c9557c77-n49l7" event={"ID":"f3011c35-3232-4fd7-81be-b178a0dad4d4","Type":"ContainerStarted","Data":"d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f"} Nov 02 22:58:37.269406 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:37.269346 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:37.378050 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:37.377991 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b67978596-nrwt6" podStartSLOduration=1.33394679 podStartE2EDuration="7.377977133s" totalImagesPullingTime="6.044030343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:37.37646823 +0000 UTC m=+310.783780061" watchObservedRunningTime="2026-11-02 22:58:37.377977133 +0000 UTC m=+310.785288963" Nov 02 22:58:37.484229 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:37.484176 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-66c9557c77-n49l7" podStartSLOduration=1.92801386 podStartE2EDuration="8.484161088s" totalImagesPullingTime="6.556147228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:37.484027924 +0000 UTC m=+310.891339753" watchObservedRunningTime="2026-11-02 22:58:37.484161088 +0000 UTC m=+310.891472912" Nov 02 22:58:40.059389 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.059353 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6"] Nov 02 22:58:40.063011 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.062987 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.114781 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.114739 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 22:58:40.127185 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.127127 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eeb11765-1103-4234-aeef-55ecae7338dd-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.127352 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.127296 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb11765-1103-4234-aeef-55ecae7338dd-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.127421 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.127372 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/eeb11765-1103-4234-aeef-55ecae7338dd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.127503 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.127480 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtg4t\" (UniqueName: \"kubernetes.io/projected/eeb11765-1103-4234-aeef-55ecae7338dd-kube-api-access-rtg4t\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.161022 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.160996 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-crzf8\"" Nov 02 22:58:40.161343 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.161322 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 22:58:40.161442 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.161381 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 22:58:40.162998 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.162980 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6"] Nov 02 22:58:40.228332 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.228294 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/eeb11765-1103-4234-aeef-55ecae7338dd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.228512 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.228355 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtg4t\" (UniqueName: \"kubernetes.io/projected/eeb11765-1103-4234-aeef-55ecae7338dd-kube-api-access-rtg4t\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.228512 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.228387 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eeb11765-1103-4234-aeef-55ecae7338dd-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.228512 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.228449 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb11765-1103-4234-aeef-55ecae7338dd-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.228512 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:58:40.228485 5672 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 22:58:40.228697 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:58:40.228516 5672 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 02 22:58:40.228697 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:58:40.228552 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eeb11765-1103-4234-aeef-55ecae7338dd-proxy-source podName:eeb11765-1103-4234-aeef-55ecae7338dd nodeName:}" failed. No retries permitted until 2026-11-02 22:58:40.728531517 +0000 UTC m=+314.135843342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/eeb11765-1103-4234-aeef-55ecae7338dd-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-knpb6" (UID: "eeb11765-1103-4234-aeef-55ecae7338dd") : configmap "usage-tenancy-proxy-source" not found Nov 02 22:58:40.228697 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:58:40.228570 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb11765-1103-4234-aeef-55ecae7338dd-tls-cert podName:eeb11765-1103-4234-aeef-55ecae7338dd nodeName:}" failed. No retries permitted until 2026-11-02 22:58:40.728563184 +0000 UTC m=+314.135874992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/eeb11765-1103-4234-aeef-55ecae7338dd-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-knpb6" (UID: "eeb11765-1103-4234-aeef-55ecae7338dd") : secret "usage-logs-tenancy-proxy-tls" not found Nov 02 22:58:40.229096 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.229080 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb11765-1103-4234-aeef-55ecae7338dd-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.283834 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.283800 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtg4t\" (UniqueName: \"kubernetes.io/projected/eeb11765-1103-4234-aeef-55ecae7338dd-kube-api-access-rtg4t\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.733607 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.733551 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/eeb11765-1103-4234-aeef-55ecae7338dd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.733811 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.733630 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eeb11765-1103-4234-aeef-55ecae7338dd-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.734441 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.734417 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/eeb11765-1103-4234-aeef-55ecae7338dd-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.735987 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.735964 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eeb11765-1103-4234-aeef-55ecae7338dd-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-knpb6\" (UID: \"eeb11765-1103-4234-aeef-55ecae7338dd\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:40.975832 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:40.975799 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:41.334904 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:41.334874 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6"] Nov 02 22:58:41.337384 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:58:41.337356 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb11765_1103_4234_aeef_55ecae7338dd.slice/crio-ffc91f2266f1a91680a967b367ae0f1df61bf4cfb7490f56efac9f9a3ebeeb8e WatchSource:0}: Error finding container ffc91f2266f1a91680a967b367ae0f1df61bf4cfb7490f56efac9f9a3ebeeb8e: Status 404 returned error can't find the container with id ffc91f2266f1a91680a967b367ae0f1df61bf4cfb7490f56efac9f9a3ebeeb8e Nov 02 22:58:42.291952 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:42.291918 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" event={"ID":"eeb11765-1103-4234-aeef-55ecae7338dd","Type":"ContainerStarted","Data":"ffc91f2266f1a91680a967b367ae0f1df61bf4cfb7490f56efac9f9a3ebeeb8e"} Nov 02 22:58:48.276085 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:48.276054 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:48.277106 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:48.277085 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 02 22:58:48.572943 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:48.572868 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66c9557c77-n49l7"] Nov 02 22:58:48.573521 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:48.573289 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-66c9557c77-n49l7" podUID="f3011c35-3232-4fd7-81be-b178a0dad4d4" containerName="manager" containerID="cri-o://d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f" gracePeriod=30 Nov 02 22:58:48.893176 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:48.892887 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:49.024706 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.024657 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-metrics-certs\") pod \"f3011c35-3232-4fd7-81be-b178a0dad4d4\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " Nov 02 22:58:49.024894 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.024738 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-webhook-cert\") pod \"f3011c35-3232-4fd7-81be-b178a0dad4d4\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " Nov 02 22:58:49.024894 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.024812 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3011c35-3232-4fd7-81be-b178a0dad4d4-kube-api-access-xxpss\" (UniqueName: \"kubernetes.io/projected/f3011c35-3232-4fd7-81be-b178a0dad4d4-kube-api-access-xxpss\") pod \"f3011c35-3232-4fd7-81be-b178a0dad4d4\" (UID: \"f3011c35-3232-4fd7-81be-b178a0dad4d4\") " Nov 02 22:58:49.034425 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.030673 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-metrics-certs" pod "f3011c35-3232-4fd7-81be-b178a0dad4d4" (UID: "f3011c35-3232-4fd7-81be-b178a0dad4d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:58:49.037077 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.037039 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-webhook-cert" pod "f3011c35-3232-4fd7-81be-b178a0dad4d4" (UID: "f3011c35-3232-4fd7-81be-b178a0dad4d4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:58:49.037416 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.037386 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3011c35-3232-4fd7-81be-b178a0dad4d4-kube-api-access-xxpss" pod "f3011c35-3232-4fd7-81be-b178a0dad4d4" (UID: "f3011c35-3232-4fd7-81be-b178a0dad4d4"). InnerVolumeSpecName "kube-api-access-xxpss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:58:49.126643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.126567 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxpss\" (UniqueName: \"kubernetes.io/projected/f3011c35-3232-4fd7-81be-b178a0dad4d4-kube-api-access-xxpss\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:58:49.126643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.126602 5672 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-metrics-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:58:49.126643 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.126617 5672 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3011c35-3232-4fd7-81be-b178a0dad4d4-webhook-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:58:49.332724 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.332689 5672 generic.go:356] "Generic (PLEG): container finished" podID="f3011c35-3232-4fd7-81be-b178a0dad4d4" containerID="d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f" exitCode=0 Nov 02 22:58:49.333112 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.332740 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c9557c77-n49l7" Nov 02 22:58:49.333112 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.332778 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c9557c77-n49l7" event={"ID":"f3011c35-3232-4fd7-81be-b178a0dad4d4","Type":"ContainerDied","Data":"d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f"} Nov 02 22:58:49.333112 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.332835 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c9557c77-n49l7" event={"ID":"f3011c35-3232-4fd7-81be-b178a0dad4d4","Type":"ContainerDied","Data":"1bc7d3b37788383cf0441f81fa646f8cab62e0145dce51a3d97fcc2b3dc90e36"} Nov 02 22:58:49.333112 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.332856 5672 scope.go:122] "RemoveContainer" containerID="d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f" Nov 02 22:58:49.345129 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.345076 5672 scope.go:122] "RemoveContainer" containerID="d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f" Nov 02 22:58:49.345506 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:58:49.345478 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f\": container with ID starting with d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f not found: ID does not exist" containerID="d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f" Nov 02 22:58:49.345588 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.345516 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f"} err="failed to get container status \"d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f\": rpc error: code = NotFound desc = could not find container \"d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f\": container with ID starting with d63944de705a72cba86f880eb978d82a1c7cc4a5846c84d07c6b53e8b86bf95f not found: ID does not exist" Nov 02 22:58:49.487495 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.487465 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66c9557c77-n49l7"] Nov 02 22:58:49.533947 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:49.533913 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66c9557c77-n49l7"] Nov 02 22:58:50.712637 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:50.712599 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3011c35-3232-4fd7-81be-b178a0dad4d4" path="/var/lib/kubelet/pods/f3011c35-3232-4fd7-81be-b178a0dad4d4/volumes" Nov 02 22:58:54.361585 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:54.361546 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" event={"ID":"eeb11765-1103-4234-aeef-55ecae7338dd","Type":"ContainerStarted","Data":"213ffba4ba7ab7599ade2b7ef4713c2e7291b83346e88a966706715549e6ae3a"} Nov 02 22:58:54.361945 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:54.361698 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:58:54.469810 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:54.469759 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" podStartSLOduration=1.636075851 podStartE2EDuration="14.469745122s" totalImagesPullingTime="12.833669271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:58:54.467926738 +0000 UTC m=+327.875238567" watchObservedRunningTime="2026-11-02 22:58:54.469745122 +0000 UTC m=+327.877056951" Nov 02 22:58:56.019619 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:58:56.019584 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:58:59.806657 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.806621 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2f5hv"] Nov 02 22:58:59.807046 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.807011 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f3011c35-3232-4fd7-81be-b178a0dad4d4" containerName="manager" Nov 02 22:58:59.807046 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.807022 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3011c35-3232-4fd7-81be-b178a0dad4d4" containerName="manager" Nov 02 22:58:59.807116 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.807093 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3011c35-3232-4fd7-81be-b178a0dad4d4" containerName="manager" Nov 02 22:58:59.810260 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.810240 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" Nov 02 22:58:59.871881 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.871857 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pp4tq\"" Nov 02 22:58:59.874408 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.874382 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2f5hv"] Nov 02 22:58:59.907091 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.907069 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 22:58:59.920994 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.920971 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj"] Nov 02 22:58:59.924353 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.924337 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 22:58:59.938499 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.938476 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrrr7\" (UniqueName: \"kubernetes.io/projected/c1428a43-3796-41a6-b8e1-79c8062885b3-kube-api-access-rrrr7\") pod \"kserve-controller-manager-776cc66d47-2f5hv\" (UID: \"c1428a43-3796-41a6-b8e1-79c8062885b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" Nov 02 22:58:59.938582 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.938535 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1428a43-3796-41a6-b8e1-79c8062885b3-cert\") pod \"kserve-controller-manager-776cc66d47-2f5hv\" (UID: \"c1428a43-3796-41a6-b8e1-79c8062885b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" Nov 02 22:58:59.975664 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.975632 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 22:58:59.975790 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.975666 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cxxpw\"" Nov 02 22:58:59.976879 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:58:59.976859 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj"] Nov 02 22:59:00.039767 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.039740 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrr7\" (UniqueName: \"kubernetes.io/projected/c1428a43-3796-41a6-b8e1-79c8062885b3-kube-api-access-rrrr7\") pod \"kserve-controller-manager-776cc66d47-2f5hv\" (UID: \"c1428a43-3796-41a6-b8e1-79c8062885b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" Nov 02 22:59:00.039892 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.039814 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2vh\" (UniqueName: \"kubernetes.io/projected/9babe17d-e87a-4811-b987-76d66d1f0990-kube-api-access-ht2vh\") pod \"llmisvc-controller-manager-869d5dcc5d-rhlfj\" (UID: \"9babe17d-e87a-4811-b987-76d66d1f0990\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 22:59:00.039892 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.039840 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1428a43-3796-41a6-b8e1-79c8062885b3-cert\") pod \"kserve-controller-manager-776cc66d47-2f5hv\" (UID: \"c1428a43-3796-41a6-b8e1-79c8062885b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" Nov 02 22:59:00.039892 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.039870 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9babe17d-e87a-4811-b987-76d66d1f0990-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rhlfj\" (UID: \"9babe17d-e87a-4811-b987-76d66d1f0990\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 22:59:00.042101 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.042075 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fg848"] Nov 02 22:59:00.042494 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.042470 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1428a43-3796-41a6-b8e1-79c8062885b3-cert\") pod \"kserve-controller-manager-776cc66d47-2f5hv\" (UID: \"c1428a43-3796-41a6-b8e1-79c8062885b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" Nov 02 22:59:00.050262 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.050244 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-fg848" Nov 02 22:59:00.093003 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.092943 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrr7\" (UniqueName: \"kubernetes.io/projected/c1428a43-3796-41a6-b8e1-79c8062885b3-kube-api-access-rrrr7\") pod \"kserve-controller-manager-776cc66d47-2f5hv\" (UID: \"c1428a43-3796-41a6-b8e1-79c8062885b3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" Nov 02 22:59:00.101791 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.101464 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fg848"] Nov 02 22:59:00.122997 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.122973 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" Nov 02 22:59:00.140455 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.140430 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2vh\" (UniqueName: \"kubernetes.io/projected/9babe17d-e87a-4811-b987-76d66d1f0990-kube-api-access-ht2vh\") pod \"llmisvc-controller-manager-869d5dcc5d-rhlfj\" (UID: \"9babe17d-e87a-4811-b987-76d66d1f0990\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 22:59:00.140589 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.140466 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e01866-fe73-471f-8e2c-f126cb04b8b8-tls-certs\") pod \"model-serving-api-75bd89fdfc-fg848\" (UID: \"30e01866-fe73-471f-8e2c-f126cb04b8b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-fg848" Nov 02 22:59:00.140589 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.140506 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hztqx\" (UniqueName: \"kubernetes.io/projected/30e01866-fe73-471f-8e2c-f126cb04b8b8-kube-api-access-hztqx\") pod \"model-serving-api-75bd89fdfc-fg848\" (UID: \"30e01866-fe73-471f-8e2c-f126cb04b8b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-fg848" Nov 02 22:59:00.140589 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.140541 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9babe17d-e87a-4811-b987-76d66d1f0990-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rhlfj\" (UID: \"9babe17d-e87a-4811-b987-76d66d1f0990\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 22:59:00.143104 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.143080 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9babe17d-e87a-4811-b987-76d66d1f0990-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rhlfj\" (UID: \"9babe17d-e87a-4811-b987-76d66d1f0990\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 22:59:00.175665 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.175630 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 22:59:00.175889 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.175865 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hb7f5\"" Nov 02 22:59:00.195132 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.194961 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2vh\" (UniqueName: \"kubernetes.io/projected/9babe17d-e87a-4811-b987-76d66d1f0990-kube-api-access-ht2vh\") pod \"llmisvc-controller-manager-869d5dcc5d-rhlfj\" (UID: \"9babe17d-e87a-4811-b987-76d66d1f0990\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 22:59:00.236432 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.236399 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 22:59:00.241689 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.241665 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e01866-fe73-471f-8e2c-f126cb04b8b8-tls-certs\") pod \"model-serving-api-75bd89fdfc-fg848\" (UID: \"30e01866-fe73-471f-8e2c-f126cb04b8b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-fg848" Nov 02 22:59:00.241816 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.241707 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hztqx\" (UniqueName: \"kubernetes.io/projected/30e01866-fe73-471f-8e2c-f126cb04b8b8-kube-api-access-hztqx\") pod \"model-serving-api-75bd89fdfc-fg848\" (UID: \"30e01866-fe73-471f-8e2c-f126cb04b8b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-fg848" Nov 02 22:59:00.244224 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.244196 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e01866-fe73-471f-8e2c-f126cb04b8b8-tls-certs\") pod \"model-serving-api-75bd89fdfc-fg848\" (UID: \"30e01866-fe73-471f-8e2c-f126cb04b8b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-fg848" Nov 02 22:59:00.296314 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.296000 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hztqx\" (UniqueName: \"kubernetes.io/projected/30e01866-fe73-471f-8e2c-f126cb04b8b8-kube-api-access-hztqx\") pod \"model-serving-api-75bd89fdfc-fg848\" (UID: \"30e01866-fe73-471f-8e2c-f126cb04b8b8\") " pod="opendatahub/model-serving-api-75bd89fdfc-fg848" Nov 02 22:59:00.362543 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.362451 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-fg848" Nov 02 22:59:00.494322 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.494295 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2f5hv"] Nov 02 22:59:00.496908 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:59:00.496873 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1428a43_3796_41a6_b8e1_79c8062885b3.slice/crio-69c2e3f5705d8acc6e1b654caa1aa440865692221a301f3a9f6d95083dd77d1f WatchSource:0}: Error finding container 69c2e3f5705d8acc6e1b654caa1aa440865692221a301f3a9f6d95083dd77d1f: Status 404 returned error can't find the container with id 69c2e3f5705d8acc6e1b654caa1aa440865692221a301f3a9f6d95083dd77d1f Nov 02 22:59:00.598384 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.598360 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj"] Nov 02 22:59:00.600535 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:59:00.600511 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9babe17d_e87a_4811_b987_76d66d1f0990.slice/crio-9ee3624c18aadae5f33be350acc98f7f664ecece1b39a60842c6919fabca2659 WatchSource:0}: Error finding container 9ee3624c18aadae5f33be350acc98f7f664ecece1b39a60842c6919fabca2659: Status 404 returned error can't find the container with id 9ee3624c18aadae5f33be350acc98f7f664ecece1b39a60842c6919fabca2659 Nov 02 22:59:00.724539 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:00.724514 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-fg848"] Nov 02 22:59:00.726415 ip-10-0-137-4 kubenswrapper[5672]: W1102 22:59:00.726389 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e01866_fe73_471f_8e2c_f126cb04b8b8.slice/crio-bb317330834af9957f7f41918b01ec993db0df89f2df61e414d7fce9c4575c04 WatchSource:0}: Error finding container bb317330834af9957f7f41918b01ec993db0df89f2df61e414d7fce9c4575c04: Status 404 returned error can't find the container with id bb317330834af9957f7f41918b01ec993db0df89f2df61e414d7fce9c4575c04 Nov 02 22:59:01.396820 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:01.396725 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-fg848" event={"ID":"30e01866-fe73-471f-8e2c-f126cb04b8b8","Type":"ContainerStarted","Data":"bb317330834af9957f7f41918b01ec993db0df89f2df61e414d7fce9c4575c04"} Nov 02 22:59:01.398051 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:01.398022 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" event={"ID":"9babe17d-e87a-4811-b987-76d66d1f0990","Type":"ContainerStarted","Data":"9ee3624c18aadae5f33be350acc98f7f664ecece1b39a60842c6919fabca2659"} Nov 02 22:59:01.399398 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:01.399377 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" event={"ID":"c1428a43-3796-41a6-b8e1-79c8062885b3","Type":"ContainerStarted","Data":"69c2e3f5705d8acc6e1b654caa1aa440865692221a301f3a9f6d95083dd77d1f"} Nov 02 22:59:03.414186 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:03.414042 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" event={"ID":"c1428a43-3796-41a6-b8e1-79c8062885b3","Type":"ContainerStarted","Data":"9cf4810333c4b3072cd2a05291ba628112ef0efb2200c8b01e9e4ab758c49d32"} Nov 02 22:59:03.414186 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:03.414122 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" Nov 02 22:59:04.420301 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:04.420259 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-fg848" event={"ID":"30e01866-fe73-471f-8e2c-f126cb04b8b8","Type":"ContainerStarted","Data":"1fb43f84a2731f8cc85067385146845d74c81926809784f88b5b70460188e29f"} Nov 02 22:59:04.420767 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:04.420437 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-fg848" Nov 02 22:59:04.421837 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:04.421815 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" event={"ID":"9babe17d-e87a-4811-b987-76d66d1f0990","Type":"ContainerStarted","Data":"af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15"} Nov 02 22:59:04.421920 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:04.421864 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 22:59:04.526843 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:04.526787 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-fg848" podStartSLOduration=1.759947233 podStartE2EDuration="4.52677255s" totalImagesPullingTime="2.766825317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:04.525704595 +0000 UTC m=+337.933016426" watchObservedRunningTime="2026-11-02 22:59:04.52677255 +0000 UTC m=+337.934084380" Nov 02 22:59:04.527711 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:04.527682 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" podStartSLOduration=2.6952790269999998 podStartE2EDuration="5.527674317s" totalImagesPullingTime="2.83239529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:03.520256946 +0000 UTC m=+336.927568776" watchObservedRunningTime="2026-11-02 22:59:04.527674317 +0000 UTC m=+337.934986147" Nov 02 22:59:05.372770 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:05.372740 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-knpb6" Nov 02 22:59:05.480358 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:05.480304 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" podStartSLOduration=3.5954083260000003 podStartE2EDuration="6.480291336s" totalImagesPullingTime="2.88488301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:04.633578383 +0000 UTC m=+338.040890212" watchObservedRunningTime="2026-11-02 22:59:05.480291336 +0000 UTC m=+338.887603174" Nov 02 22:59:15.430708 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:15.430679 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-fg848" Nov 02 22:59:19.058529 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:19.058491 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 22:59:26.021555 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:59:26.021508 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 22:59:28.569485 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:28.569445 5672 generic.go:356] "Generic (PLEG): container finished" podID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerID="3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727" exitCode=0 Nov 02 22:59:28.569950 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:28.569579 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" event={"ID":"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b","Type":"ContainerDied","Data":"3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727"} Nov 02 22:59:28.569993 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:28.569952 5672 scope.go:122] "RemoveContainer" containerID="3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727" Nov 02 22:59:28.571112 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:28.571083 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:59:28.571193 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:28.571173 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:59:28.571232 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:28.571210 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:59:29.575992 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:29.575953 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" event={"ID":"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b","Type":"ContainerStarted","Data":"457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7"} Nov 02 22:59:29.892245 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:29.892164 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:59:29.892393 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:29.892315 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7jln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 02 22:59:29.892393 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:29.892372 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 02 22:59:30.892435 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:30.892398 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7jln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 02 22:59:30.892821 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:30.892449 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 02 22:59:31.892305 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:31.892268 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7jln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 02 22:59:31.892536 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:31.892337 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 02 22:59:32.892409 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:32.892369 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7jln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 02 22:59:32.892611 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:32.892435 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 02 22:59:33.892290 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:33.892257 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7jln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 02 22:59:33.892457 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:33.892311 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 02 22:59:34.431174 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:34.429419 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-2f5hv" Nov 02 22:59:34.896078 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:34.896046 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:59:34.896420 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:34.896403 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:59:34.897051 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:34.897029 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 22:59:35.428902 ip-10-0-137-4 kubenswrapper[5672]: I1102 22:59:35.428876 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 22:59:56.019587 ip-10-0-137-4 kubenswrapper[5672]: E1102 22:59:56.019547 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:00:00.155193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.155158 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7"] Nov 02 23:00:00.157911 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.157894 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" Nov 02 23:00:00.209764 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.209728 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:00:00.211815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.211790 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7"] Nov 02 23:00:00.212365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.212339 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f00d02-5426-41b7-a93f-865e6fa643c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-vwvb7\" (UID: \"a3f00d02-5426-41b7-a93f-865e6fa643c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" Nov 02 23:00:00.212486 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.212468 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8hh\" (UniqueName: \"kubernetes.io/projected/a3f00d02-5426-41b7-a93f-865e6fa643c2-kube-api-access-jx8hh\") pod \"maas-api-key-cleanup-29894340-vwvb7\" (UID: \"a3f00d02-5426-41b7-a93f-865e6fa643c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" Nov 02 23:00:00.256099 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.256071 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:00:00.256246 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.256107 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:00:00.313323 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.313296 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8hh\" (UniqueName: \"kubernetes.io/projected/a3f00d02-5426-41b7-a93f-865e6fa643c2-kube-api-access-jx8hh\") pod \"maas-api-key-cleanup-29894340-vwvb7\" (UID: \"a3f00d02-5426-41b7-a93f-865e6fa643c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" Nov 02 23:00:00.313467 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.313336 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f00d02-5426-41b7-a93f-865e6fa643c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-vwvb7\" (UID: \"a3f00d02-5426-41b7-a93f-865e6fa643c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" Nov 02 23:00:00.313955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.313936 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f00d02-5426-41b7-a93f-865e6fa643c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894340-vwvb7\" (UID: \"a3f00d02-5426-41b7-a93f-865e6fa643c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" Nov 02 23:00:00.366565 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.366537 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8hh\" (UniqueName: \"kubernetes.io/projected/a3f00d02-5426-41b7-a93f-865e6fa643c2-kube-api-access-jx8hh\") pod \"maas-api-key-cleanup-29894340-vwvb7\" (UID: \"a3f00d02-5426-41b7-a93f-865e6fa643c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" Nov 02 23:00:00.469211 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.469186 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" Nov 02 23:00:00.831724 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:00.831639 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7"] Nov 02 23:00:00.834730 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:00:00.834691 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f00d02_5426_41b7_a93f_865e6fa643c2.slice/crio-28d9cc81517566bc4f537bfa9630349edf4ebc0eb11a0f6017bf8b559e75b610 WatchSource:0}: Error finding container 28d9cc81517566bc4f537bfa9630349edf4ebc0eb11a0f6017bf8b559e75b610: Status 404 returned error can't find the container with id 28d9cc81517566bc4f537bfa9630349edf4ebc0eb11a0f6017bf8b559e75b610 Nov 02 23:00:01.725871 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:01.725835 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" event={"ID":"a3f00d02-5426-41b7-a93f-865e6fa643c2","Type":"ContainerStarted","Data":"28d9cc81517566bc4f537bfa9630349edf4ebc0eb11a0f6017bf8b559e75b610"} Nov 02 23:00:09.764922 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:09.764888 5672 generic.go:356] "Generic (PLEG): container finished" podID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerID="afe338669b79571b5765ca770a64ba667702adeb7cb56359c7d61caafc997d5c" exitCode=60 Nov 02 23:00:09.765368 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:09.764966 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" event={"ID":"a3f00d02-5426-41b7-a93f-865e6fa643c2","Type":"ContainerDied","Data":"afe338669b79571b5765ca770a64ba667702adeb7cb56359c7d61caafc997d5c"} Nov 02 23:00:09.765368 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:09.765209 5672 scope.go:122] "RemoveContainer" containerID="afe338669b79571b5765ca770a64ba667702adeb7cb56359c7d61caafc997d5c" Nov 02 23:00:10.771683 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:10.771655 5672 generic.go:356] "Generic (PLEG): container finished" podID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerID="e43f59d94b3a03470c08bb638e524111e5ace6b8dea8e69646ad62573818fe7a" exitCode=60 Nov 02 23:00:10.772084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:10.771744 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" event={"ID":"a3f00d02-5426-41b7-a93f-865e6fa643c2","Type":"ContainerDied","Data":"e43f59d94b3a03470c08bb638e524111e5ace6b8dea8e69646ad62573818fe7a"} Nov 02 23:00:10.772084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:10.771799 5672 scope.go:122] "RemoveContainer" containerID="afe338669b79571b5765ca770a64ba667702adeb7cb56359c7d61caafc997d5c" Nov 02 23:00:10.772084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:10.771910 5672 scope.go:122] "RemoveContainer" containerID="e43f59d94b3a03470c08bb638e524111e5ace6b8dea8e69646ad62573818fe7a" Nov 02 23:00:10.772236 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:00:10.772111 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-vwvb7_odh-ai-gateway-infra(a3f00d02-5426-41b7-a93f-865e6fa643c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" Nov 02 23:00:11.778141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:11.778113 5672 scope.go:122] "RemoveContainer" containerID="e43f59d94b3a03470c08bb638e524111e5ace6b8dea8e69646ad62573818fe7a" Nov 02 23:00:11.778536 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:00:11.778336 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-vwvb7_odh-ai-gateway-infra(a3f00d02-5426-41b7-a93f-865e6fa643c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" Nov 02 23:00:16.623465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:16.623431 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc"] Nov 02 23:00:16.628057 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:16.628038 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" Nov 02 23:00:16.681030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:16.681002 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc"] Nov 02 23:00:16.762204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:16.762173 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db028e9f-d723-4a0c-998e-e1712a86b183-cert\") pod \"llmisvc-controller-manager-6466c554d8-gt9zc\" (UID: \"db028e9f-d723-4a0c-998e-e1712a86b183\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" Nov 02 23:00:16.762367 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:16.762219 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s84b\" (UniqueName: \"kubernetes.io/projected/db028e9f-d723-4a0c-998e-e1712a86b183-kube-api-access-7s84b\") pod \"llmisvc-controller-manager-6466c554d8-gt9zc\" (UID: \"db028e9f-d723-4a0c-998e-e1712a86b183\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" Nov 02 23:00:16.863478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:16.863445 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s84b\" (UniqueName: \"kubernetes.io/projected/db028e9f-d723-4a0c-998e-e1712a86b183-kube-api-access-7s84b\") pod \"llmisvc-controller-manager-6466c554d8-gt9zc\" (UID: \"db028e9f-d723-4a0c-998e-e1712a86b183\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" Nov 02 23:00:16.863635 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:16.863522 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db028e9f-d723-4a0c-998e-e1712a86b183-cert\") pod \"llmisvc-controller-manager-6466c554d8-gt9zc\" (UID: \"db028e9f-d723-4a0c-998e-e1712a86b183\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" Nov 02 23:00:16.865836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:16.865816 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db028e9f-d723-4a0c-998e-e1712a86b183-cert\") pod \"llmisvc-controller-manager-6466c554d8-gt9zc\" (UID: \"db028e9f-d723-4a0c-998e-e1712a86b183\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" Nov 02 23:00:16.916855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:16.916790 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s84b\" (UniqueName: \"kubernetes.io/projected/db028e9f-d723-4a0c-998e-e1712a86b183-kube-api-access-7s84b\") pod \"llmisvc-controller-manager-6466c554d8-gt9zc\" (UID: \"db028e9f-d723-4a0c-998e-e1712a86b183\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" Nov 02 23:00:16.939944 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:16.939922 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" Nov 02 23:00:17.300742 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:17.300719 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc"] Nov 02 23:00:17.303024 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:00:17.302993 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb028e9f_d723_4a0c_998e_e1712a86b183.slice/crio-8e240370d4473a1337b2637da1c491d8d36c3c49ccf625faa97b941f3ef4f928 WatchSource:0}: Error finding container 8e240370d4473a1337b2637da1c491d8d36c3c49ccf625faa97b941f3ef4f928: Status 404 returned error can't find the container with id 8e240370d4473a1337b2637da1c491d8d36c3c49ccf625faa97b941f3ef4f928 Nov 02 23:00:17.806974 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:17.806940 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" event={"ID":"db028e9f-d723-4a0c-998e-e1712a86b183","Type":"ContainerStarted","Data":"8e240370d4473a1337b2637da1c491d8d36c3c49ccf625faa97b941f3ef4f928"} Nov 02 23:00:18.812813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:18.812768 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" event={"ID":"db028e9f-d723-4a0c-998e-e1712a86b183","Type":"ContainerStarted","Data":"1c909132b38094864c0c972bece0d0263380158adf98bbf1d345eada8a4794d8"} Nov 02 23:00:18.813325 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:18.812905 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" Nov 02 23:00:18.918349 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:18.918292 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" podStartSLOduration=2.137053773 podStartE2EDuration="2.918277056s" totalImagesPullingTime="781.223283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:18.917772986 +0000 UTC m=+412.325084817" watchObservedRunningTime="2026-11-02 23:00:18.918277056 +0000 UTC m=+412.325588887" Nov 02 23:00:22.705886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:22.705857 5672 scope.go:122] "RemoveContainer" containerID="e43f59d94b3a03470c08bb638e524111e5ace6b8dea8e69646ad62573818fe7a" Nov 02 23:00:23.839134 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:23.839046 5672 generic.go:356] "Generic (PLEG): container finished" podID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerID="cc7624c474a3909280b0371c282ddd88a9e2d9f94cf7ec7b94015a32222bb355" exitCode=60 Nov 02 23:00:23.839134 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:23.839117 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" event={"ID":"a3f00d02-5426-41b7-a93f-865e6fa643c2","Type":"ContainerDied","Data":"cc7624c474a3909280b0371c282ddd88a9e2d9f94cf7ec7b94015a32222bb355"} Nov 02 23:00:23.839648 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:23.839205 5672 scope.go:122] "RemoveContainer" containerID="e43f59d94b3a03470c08bb638e524111e5ace6b8dea8e69646ad62573818fe7a" Nov 02 23:00:23.839648 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:23.839547 5672 scope.go:122] "RemoveContainer" containerID="cc7624c474a3909280b0371c282ddd88a9e2d9f94cf7ec7b94015a32222bb355" Nov 02 23:00:23.839831 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:00:23.839810 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894340-vwvb7_odh-ai-gateway-infra(a3f00d02-5426-41b7-a93f-865e6fa643c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" Nov 02 23:00:24.955283 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:24.955250 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7"] Nov 02 23:00:25.082903 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:25.082879 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" Nov 02 23:00:25.237653 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:25.237624 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f00d02-5426-41b7-a93f-865e6fa643c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f00d02-5426-41b7-a93f-865e6fa643c2-maas-api-service-ca\") pod \"a3f00d02-5426-41b7-a93f-865e6fa643c2\" (UID: \"a3f00d02-5426-41b7-a93f-865e6fa643c2\") " Nov 02 23:00:25.237822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:25.237661 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f00d02-5426-41b7-a93f-865e6fa643c2-kube-api-access-jx8hh\" (UniqueName: \"kubernetes.io/projected/a3f00d02-5426-41b7-a93f-865e6fa643c2-kube-api-access-jx8hh\") pod \"a3f00d02-5426-41b7-a93f-865e6fa643c2\" (UID: \"a3f00d02-5426-41b7-a93f-865e6fa643c2\") " Nov 02 23:00:25.237998 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:25.237975 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f00d02-5426-41b7-a93f-865e6fa643c2-maas-api-service-ca" pod "a3f00d02-5426-41b7-a93f-865e6fa643c2" (UID: "a3f00d02-5426-41b7-a93f-865e6fa643c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:00:25.240109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:25.240077 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f00d02-5426-41b7-a93f-865e6fa643c2-kube-api-access-jx8hh" pod "a3f00d02-5426-41b7-a93f-865e6fa643c2" (UID: "a3f00d02-5426-41b7-a93f-865e6fa643c2"). InnerVolumeSpecName "kube-api-access-jx8hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:00:25.338790 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:25.338758 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f00d02-5426-41b7-a93f-865e6fa643c2-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:00:25.338790 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:25.338786 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx8hh\" (UniqueName: \"kubernetes.io/projected/a3f00d02-5426-41b7-a93f-865e6fa643c2-kube-api-access-jx8hh\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:00:25.852877 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:25.852838 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" event={"ID":"a3f00d02-5426-41b7-a93f-865e6fa643c2","Type":"ContainerDied","Data":"28d9cc81517566bc4f537bfa9630349edf4ebc0eb11a0f6017bf8b559e75b610"} Nov 02 23:00:25.853052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:25.852895 5672 scope.go:122] "RemoveContainer" containerID="cc7624c474a3909280b0371c282ddd88a9e2d9f94cf7ec7b94015a32222bb355" Nov 02 23:00:25.853052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:25.852853 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7" Nov 02 23:00:26.004588 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:26.004562 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7"] Nov 02 23:00:26.019868 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:00:26.019835 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:00:26.050975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:26.050947 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894340-vwvb7"] Nov 02 23:00:26.718092 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:26.718064 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" path="/var/lib/kubelet/pods/a3f00d02-5426-41b7-a93f-865e6fa643c2/volumes" Nov 02 23:00:49.819544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:49.819509 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gt9zc" Nov 02 23:00:49.973949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:49.973916 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj"] Nov 02 23:00:49.974186 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:49.974163 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" podUID="9babe17d-e87a-4811-b987-76d66d1f0990" containerName="manager" containerID="cri-o://af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15" gracePeriod=30 Nov 02 23:00:50.213405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.213382 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 23:00:50.249338 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.249315 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9babe17d-e87a-4811-b987-76d66d1f0990-kube-api-access-ht2vh\" (UniqueName: \"kubernetes.io/projected/9babe17d-e87a-4811-b987-76d66d1f0990-kube-api-access-ht2vh\") pod \"9babe17d-e87a-4811-b987-76d66d1f0990\" (UID: \"9babe17d-e87a-4811-b987-76d66d1f0990\") " Nov 02 23:00:50.249437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.249398 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9babe17d-e87a-4811-b987-76d66d1f0990-cert\" (UniqueName: \"kubernetes.io/secret/9babe17d-e87a-4811-b987-76d66d1f0990-cert\") pod \"9babe17d-e87a-4811-b987-76d66d1f0990\" (UID: \"9babe17d-e87a-4811-b987-76d66d1f0990\") " Nov 02 23:00:50.251287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.251262 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9babe17d-e87a-4811-b987-76d66d1f0990-kube-api-access-ht2vh" pod "9babe17d-e87a-4811-b987-76d66d1f0990" (UID: "9babe17d-e87a-4811-b987-76d66d1f0990"). InnerVolumeSpecName "kube-api-access-ht2vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:00:50.251380 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.251335 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9babe17d-e87a-4811-b987-76d66d1f0990-cert" pod "9babe17d-e87a-4811-b987-76d66d1f0990" (UID: "9babe17d-e87a-4811-b987-76d66d1f0990"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:00:50.350804 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.350735 5672 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9babe17d-e87a-4811-b987-76d66d1f0990-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:00:50.350804 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.350762 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht2vh\" (UniqueName: \"kubernetes.io/projected/9babe17d-e87a-4811-b987-76d66d1f0990-kube-api-access-ht2vh\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:00:50.971825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.971748 5672 generic.go:356] "Generic (PLEG): container finished" podID="9babe17d-e87a-4811-b987-76d66d1f0990" containerID="af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15" exitCode=0 Nov 02 23:00:50.971825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.971817 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" Nov 02 23:00:50.972240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.971828 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" event={"ID":"9babe17d-e87a-4811-b987-76d66d1f0990","Type":"ContainerDied","Data":"af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15"} Nov 02 23:00:50.972240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.971917 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj" event={"ID":"9babe17d-e87a-4811-b987-76d66d1f0990","Type":"ContainerDied","Data":"9ee3624c18aadae5f33be350acc98f7f664ecece1b39a60842c6919fabca2659"} Nov 02 23:00:50.972240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.971939 5672 scope.go:122] "RemoveContainer" containerID="af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15" Nov 02 23:00:50.980544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.980379 5672 scope.go:122] "RemoveContainer" containerID="af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15" Nov 02 23:00:50.980660 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:00:50.980639 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15\": container with ID starting with af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15 not found: ID does not exist" containerID="af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15" Nov 02 23:00:50.980716 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:50.980672 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15"} err="failed to get container status \"af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15\": rpc error: code = NotFound desc = could not find container \"af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15\": container with ID starting with af88e8305497991df56d4b15ade3bdc79a5c2a3e031542d343b40ff93227fb15 not found: ID does not exist" Nov 02 23:00:51.122864 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:51.122834 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj"] Nov 02 23:00:51.126846 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:51.126822 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rhlfj"] Nov 02 23:00:52.711089 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:00:52.711059 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9babe17d-e87a-4811-b987-76d66d1f0990" path="/var/lib/kubelet/pods/9babe17d-e87a-4811-b987-76d66d1f0990/volumes" Nov 02 23:00:56.020843 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:00:56.020782 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:01:26.020826 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:01:26.020745 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:01:32.010817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:01:32.010778 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:01:43.331261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:01:43.331223 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:01:54.125858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:01:54.125817 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:01:56.020644 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:01:56.020604 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:02:10.403322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:10.403284 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:02:26.026141 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:02:26.026089 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:02:27.626358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.626315 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh"] Nov 02 23:02:27.626883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.626857 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerName="cleanup" Nov 02 23:02:27.626960 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.626887 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerName="cleanup" Nov 02 23:02:27.626960 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.626902 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerName="cleanup" Nov 02 23:02:27.626960 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.626909 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerName="cleanup" Nov 02 23:02:27.626960 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.626953 5672 cpu_manager.go:414] "containerMap: removing container" podUID="9babe17d-e87a-4811-b987-76d66d1f0990" containerName="manager" Nov 02 23:02:27.627176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.626961 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9babe17d-e87a-4811-b987-76d66d1f0990" containerName="manager" Nov 02 23:02:27.627176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.627065 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerName="cleanup" Nov 02 23:02:27.627176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.627075 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerName="cleanup" Nov 02 23:02:27.627176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.627087 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerName="cleanup" Nov 02 23:02:27.627176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.627099 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="9babe17d-e87a-4811-b987-76d66d1f0990" containerName="manager" Nov 02 23:02:27.629895 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.629872 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.726808 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.726771 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh"] Nov 02 23:02:27.751608 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.751565 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.751771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.751619 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.751771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.751725 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.751771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.751766 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.751869 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.751790 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.751928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.751906 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.751963 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.751944 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vr8f\" (UniqueName: \"kubernetes.io/projected/b303eca0-a0b3-48c4-928e-23f3266ae2bc-kube-api-access-8vr8f\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.752008 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.751995 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.752043 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.752021 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.853434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.853396 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.853643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.853454 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.853643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.853486 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.853643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.853507 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.853643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.853534 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.853643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.853565 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vr8f\" (UniqueName: \"kubernetes.io/projected/b303eca0-a0b3-48c4-928e-23f3266ae2bc-kube-api-access-8vr8f\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.853643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.853597 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.853643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.853626 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.854000 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.853669 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.854181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.854132 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.854349 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.854185 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.854349 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.854212 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.854519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.854264 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.854762 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.854737 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.856030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.856005 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.856185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.856169 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.907087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.907012 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b303eca0-a0b3-48c4-928e-23f3266ae2bc-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.907246 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.907175 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vr8f\" (UniqueName: \"kubernetes.io/projected/b303eca0-a0b3-48c4-928e-23f3266ae2bc-kube-api-access-8vr8f\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2nshh\" (UID: \"b303eca0-a0b3-48c4-928e-23f3266ae2bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:27.943797 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:27.943761 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:28.360823 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:28.360797 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh"] Nov 02 23:02:28.363506 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:02:28.363477 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb303eca0_a0b3_48c4_928e_23f3266ae2bc.slice/crio-726ffc83ba208ca7bf33d00e7f6bab34106825971f3fa2c494916c879c1e4017 WatchSource:0}: Error finding container 726ffc83ba208ca7bf33d00e7f6bab34106825971f3fa2c494916c879c1e4017: Status 404 returned error can't find the container with id 726ffc83ba208ca7bf33d00e7f6bab34106825971f3fa2c494916c879c1e4017 Nov 02 23:02:28.365712 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:28.365678 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:02:28.365807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:28.365745 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:02:28.365807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:28.365776 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:02:28.464686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:28.464654 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" event={"ID":"b303eca0-a0b3-48c4-928e-23f3266ae2bc","Type":"ContainerStarted","Data":"79e20ff73911235561846800be9f78049bf4f1a93c075bfb5214eeddfa66ee57"} Nov 02 23:02:28.464857 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:28.464711 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" event={"ID":"b303eca0-a0b3-48c4-928e-23f3266ae2bc","Type":"ContainerStarted","Data":"726ffc83ba208ca7bf33d00e7f6bab34106825971f3fa2c494916c879c1e4017"} Nov 02 23:02:28.576093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:28.576031 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podStartSLOduration=1.5760110250000001 podStartE2EDuration="1.576011025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:28.574383285 +0000 UTC m=+541.981695116" watchObservedRunningTime="2026-11-02 23:02:28.576011025 +0000 UTC m=+541.983322856" Nov 02 23:02:28.948276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:28.948234 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:29.949291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:29.949250 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 23:02:29.949762 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:29.949311 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 23:02:30.950556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:30.950504 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 23:02:30.951070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:30.950581 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 23:02:31.944607 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:31.944565 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 23:02:31.944953 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:31.944911 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 23:02:32.945171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:32.944590 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 23:02:32.945171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:32.944669 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 23:02:33.946183 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:33.945674 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 23:02:33.946183 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:33.945771 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 23:02:34.944996 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:34.944957 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 23:02:34.945164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:34.945016 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 23:02:36.685897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:36.685800 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:36.686390 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:36.686214 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:36.687170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:02:36.687138 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" Nov 02 23:02:56.023730 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:02:56.023689 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:03:10.506227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:10.506191 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:03:26.020013 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:03:26.019971 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:03:28.815526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:28.815493 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:03:35.821069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.821035 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz"] Nov 02 23:03:35.821503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.821441 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerName="cleanup" Nov 02 23:03:35.821503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.821454 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f00d02-5426-41b7-a93f-865e6fa643c2" containerName="cleanup" Nov 02 23:03:35.825042 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.825018 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:35.921645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.921603 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:03:35.921645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.921637 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:03:35.921919 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.921652 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:03:35.921919 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.921644 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:03:35.923190 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.923164 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz"] Nov 02 23:03:35.994267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.994218 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mfs\" (UniqueName: \"kubernetes.io/projected/f1fef8f9-5a26-417d-8bce-03411fb862c6-kube-api-access-n7mfs\") pod \"maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:35.994471 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.994307 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:35.994543 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:35.994469 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:36.095120 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:36.095037 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mfs\" (UniqueName: \"kubernetes.io/projected/f1fef8f9-5a26-417d-8bce-03411fb862c6-kube-api-access-n7mfs\") pod \"maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:36.095120 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:36.095085 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:36.095357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:36.095182 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:36.095886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:36.095865 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:36.097645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:36.097622 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:36.148416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:36.148383 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mfs\" (UniqueName: \"kubernetes.io/projected/f1fef8f9-5a26-417d-8bce-03411fb862c6-kube-api-access-n7mfs\") pod \"maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:36.439014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:36.438939 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:36.456242 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:36.456211 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:03:36.800487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:36.800456 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz"] Nov 02 23:03:36.802806 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:03:36.802779 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fef8f9_5a26_417d_8bce_03411fb862c6.slice/crio-8df5b094cfccfde80a919cc15407fecd4883c8d2390de045adbbe4a567d21150 WatchSource:0}: Error finding container 8df5b094cfccfde80a919cc15407fecd4883c8d2390de045adbbe4a567d21150: Status 404 returned error can't find the container with id 8df5b094cfccfde80a919cc15407fecd4883c8d2390de045adbbe4a567d21150 Nov 02 23:03:36.816956 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:36.816927 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" event={"ID":"f1fef8f9-5a26-417d-8bce-03411fb862c6","Type":"ContainerStarted","Data":"8df5b094cfccfde80a919cc15407fecd4883c8d2390de045adbbe4a567d21150"} Nov 02 23:03:37.823357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:37.823316 5672 generic.go:356] "Generic (PLEG): container finished" podID="f1fef8f9-5a26-417d-8bce-03411fb862c6" containerID="3bf54c5cb17c676cece9f8ac3cd3493ffe6422f87810b78c3ce3c7fc54db027d" exitCode=0 Nov 02 23:03:37.823721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:37.823373 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" event={"ID":"f1fef8f9-5a26-417d-8bce-03411fb862c6","Type":"ContainerDied","Data":"3bf54c5cb17c676cece9f8ac3cd3493ffe6422f87810b78c3ce3c7fc54db027d"} Nov 02 23:03:38.965180 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:38.965129 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:39.124095 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.124015 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-xks-ca\") pod \"f1fef8f9-5a26-417d-8bce-03411fb862c6\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " Nov 02 23:03:39.124095 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.124077 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fef8f9-5a26-417d-8bce-03411fb862c6-kube-api-access-n7mfs\" (UniqueName: \"kubernetes.io/projected/f1fef8f9-5a26-417d-8bce-03411fb862c6-kube-api-access-n7mfs\") pod \"f1fef8f9-5a26-417d-8bce-03411fb862c6\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " Nov 02 23:03:39.124357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.124254 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-openshift-ca\") pod \"f1fef8f9-5a26-417d-8bce-03411fb862c6\" (UID: \"f1fef8f9-5a26-417d-8bce-03411fb862c6\") " Nov 02 23:03:39.124625 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.124596 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-openshift-ca" pod "f1fef8f9-5a26-417d-8bce-03411fb862c6" (UID: "f1fef8f9-5a26-417d-8bce-03411fb862c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:03:39.126221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.126192 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-xks-ca" pod "f1fef8f9-5a26-417d-8bce-03411fb862c6" (UID: "f1fef8f9-5a26-417d-8bce-03411fb862c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:03:39.126357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.126332 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fef8f9-5a26-417d-8bce-03411fb862c6-kube-api-access-n7mfs" pod "f1fef8f9-5a26-417d-8bce-03411fb862c6" (UID: "f1fef8f9-5a26-417d-8bce-03411fb862c6"). InnerVolumeSpecName "kube-api-access-n7mfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:03:39.225327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.225296 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:39.225327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.225321 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mfs\" (UniqueName: \"kubernetes.io/projected/f1fef8f9-5a26-417d-8bce-03411fb862c6-kube-api-access-n7mfs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:39.225327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.225332 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fef8f9-5a26-417d-8bce-03411fb862c6-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:39.840796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.840767 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" Nov 02 23:03:39.840964 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.840796 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz" event={"ID":"f1fef8f9-5a26-417d-8bce-03411fb862c6","Type":"ContainerDied","Data":"8df5b094cfccfde80a919cc15407fecd4883c8d2390de045adbbe4a567d21150"} Nov 02 23:03:39.840964 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:39.840867 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8df5b094cfccfde80a919cc15407fecd4883c8d2390de045adbbe4a567d21150" Nov 02 23:03:48.776520 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.776475 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb"] Nov 02 23:03:48.777135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.777080 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f1fef8f9-5a26-417d-8bce-03411fb862c6" containerName="revoke-keys" Nov 02 23:03:48.777135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.777098 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fef8f9-5a26-417d-8bce-03411fb862c6" containerName="revoke-keys" Nov 02 23:03:48.777302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.777213 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fef8f9-5a26-417d-8bce-03411fb862c6" containerName="revoke-keys" Nov 02 23:03:48.780905 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.780879 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:48.809410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.809372 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0078993-kwjbb\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:48.809579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.809515 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0078993-kwjbb\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:48.809579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.809539 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkwbk\" (UniqueName: \"kubernetes.io/projected/d086d657-6a6e-49a1-be0d-79f7322dd334-kube-api-access-gkwbk\") pod \"maas-api-revoke-keys-e2e-ait-d0078993-kwjbb\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:48.833203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.833171 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:03:48.879266 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.879222 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:03:48.879266 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.879239 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:03:48.879472 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.879291 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:03:48.880969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.880946 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb"] Nov 02 23:03:48.910693 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.910662 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0078993-kwjbb\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:48.910888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.910746 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0078993-kwjbb\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:48.910888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.910774 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkwbk\" (UniqueName: \"kubernetes.io/projected/d086d657-6a6e-49a1-be0d-79f7322dd334-kube-api-access-gkwbk\") pod \"maas-api-revoke-keys-e2e-ait-d0078993-kwjbb\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:48.911505 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.911480 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0078993-kwjbb\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:48.913059 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.913039 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0078993-kwjbb\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:48.963949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:48.963914 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkwbk\" (UniqueName: \"kubernetes.io/projected/d086d657-6a6e-49a1-be0d-79f7322dd334-kube-api-access-gkwbk\") pod \"maas-api-revoke-keys-e2e-ait-d0078993-kwjbb\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:49.094319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:49.094222 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:49.457901 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:49.457875 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb"] Nov 02 23:03:49.460186 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:03:49.460001 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd086d657_6a6e_49a1_be0d_79f7322dd334.slice/crio-15808a0fa4c522f4c5d2da43cbe558ebd50a54129d185aa170f4aa966cc55d6f WatchSource:0}: Error finding container 15808a0fa4c522f4c5d2da43cbe558ebd50a54129d185aa170f4aa966cc55d6f: Status 404 returned error can't find the container with id 15808a0fa4c522f4c5d2da43cbe558ebd50a54129d185aa170f4aa966cc55d6f Nov 02 23:03:49.902912 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:49.902874 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" event={"ID":"d086d657-6a6e-49a1-be0d-79f7322dd334","Type":"ContainerStarted","Data":"e912498d02bd9ad372cc4f877854c651e9f233ffe99634ffda1e2d25a656279b"} Nov 02 23:03:49.903332 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:49.902927 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" event={"ID":"d086d657-6a6e-49a1-be0d-79f7322dd334","Type":"ContainerStarted","Data":"15808a0fa4c522f4c5d2da43cbe558ebd50a54129d185aa170f4aa966cc55d6f"} Nov 02 23:03:50.010959 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:50.010848 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" podStartSLOduration=2.010834124 podStartE2EDuration="2.010834124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:50.009368054 +0000 UTC m=+623.416679878" watchObservedRunningTime="2026-11-02 23:03:50.010834124 +0000 UTC m=+623.418145950" Nov 02 23:03:50.909741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:50.909703 5672 generic.go:356] "Generic (PLEG): container finished" podID="d086d657-6a6e-49a1-be0d-79f7322dd334" containerID="e912498d02bd9ad372cc4f877854c651e9f233ffe99634ffda1e2d25a656279b" exitCode=0 Nov 02 23:03:50.910176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:50.909793 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" event={"ID":"d086d657-6a6e-49a1-be0d-79f7322dd334","Type":"ContainerDied","Data":"e912498d02bd9ad372cc4f877854c651e9f233ffe99634ffda1e2d25a656279b"} Nov 02 23:03:52.053699 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.053670 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:52.145227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.145190 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d086d657-6a6e-49a1-be0d-79f7322dd334-kube-api-access-gkwbk\" (UniqueName: \"kubernetes.io/projected/d086d657-6a6e-49a1-be0d-79f7322dd334-kube-api-access-gkwbk\") pod \"d086d657-6a6e-49a1-be0d-79f7322dd334\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " Nov 02 23:03:52.145426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.145273 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-xks-ca\") pod \"d086d657-6a6e-49a1-be0d-79f7322dd334\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " Nov 02 23:03:52.145426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.145315 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-openshift-ca\") pod \"d086d657-6a6e-49a1-be0d-79f7322dd334\" (UID: \"d086d657-6a6e-49a1-be0d-79f7322dd334\") " Nov 02 23:03:52.145816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.145779 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-openshift-ca" pod "d086d657-6a6e-49a1-be0d-79f7322dd334" (UID: "d086d657-6a6e-49a1-be0d-79f7322dd334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:03:52.147678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.147649 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-xks-ca" pod "d086d657-6a6e-49a1-be0d-79f7322dd334" (UID: "d086d657-6a6e-49a1-be0d-79f7322dd334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:03:52.147750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.147722 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d086d657-6a6e-49a1-be0d-79f7322dd334-kube-api-access-gkwbk" pod "d086d657-6a6e-49a1-be0d-79f7322dd334" (UID: "d086d657-6a6e-49a1-be0d-79f7322dd334"). InnerVolumeSpecName "kube-api-access-gkwbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:03:52.246743 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.246702 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:52.246743 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.246741 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkwbk\" (UniqueName: \"kubernetes.io/projected/d086d657-6a6e-49a1-be0d-79f7322dd334-kube-api-access-gkwbk\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:52.246975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.246760 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d086d657-6a6e-49a1-be0d-79f7322dd334-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:52.626418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.626331 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:03:52.920776 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.920693 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" event={"ID":"d086d657-6a6e-49a1-be0d-79f7322dd334","Type":"ContainerDied","Data":"15808a0fa4c522f4c5d2da43cbe558ebd50a54129d185aa170f4aa966cc55d6f"} Nov 02 23:03:52.920776 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.920723 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb" Nov 02 23:03:52.920776 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:52.920743 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15808a0fa4c522f4c5d2da43cbe558ebd50a54129d185aa170f4aa966cc55d6f" Nov 02 23:03:55.913342 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:55.913306 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk"] Nov 02 23:03:55.913744 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:55.913685 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d086d657-6a6e-49a1-be0d-79f7322dd334" containerName="revoke-keys" Nov 02 23:03:55.913744 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:55.913696 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d086d657-6a6e-49a1-be0d-79f7322dd334" containerName="revoke-keys" Nov 02 23:03:55.913831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:55.913788 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d086d657-6a6e-49a1-be0d-79f7322dd334" containerName="revoke-keys" Nov 02 23:03:55.917112 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:55.917092 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:55.981018 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:55.980983 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:55.981018 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:55.981024 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd6kj\" (UniqueName: \"kubernetes.io/projected/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-kube-api-access-gd6kj\") pod \"maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:55.981264 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:55.981190 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:56.014740 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.014680 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:03:56.014740 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.014704 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:03:56.014740 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.014680 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:03:56.015076 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.014933 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:03:56.017081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.017044 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk"] Nov 02 23:03:56.022845 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:03:56.022818 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:03:56.082776 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.082734 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:56.082776 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.082775 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd6kj\" (UniqueName: \"kubernetes.io/projected/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-kube-api-access-gd6kj\") pod \"maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:56.082995 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.082847 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:56.083551 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.083527 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:56.085248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.085223 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:56.136074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.136039 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd6kj\" (UniqueName: \"kubernetes.io/projected/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-kube-api-access-gd6kj\") pod \"maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:56.229302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.229254 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:56.595685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.595575 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk"] Nov 02 23:03:56.598400 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:03:56.598364 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f23f148_51d0_48be_83eb_ec6d2d1aefd3.slice/crio-1608930d13e04d67fb94a5485554e2dac2a4f25c0bf4ab88995a7311e88d5a40 WatchSource:0}: Error finding container 1608930d13e04d67fb94a5485554e2dac2a4f25c0bf4ab88995a7311e88d5a40: Status 404 returned error can't find the container with id 1608930d13e04d67fb94a5485554e2dac2a4f25c0bf4ab88995a7311e88d5a40 Nov 02 23:03:56.943847 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:56.943813 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" event={"ID":"3f23f148-51d0-48be-83eb-ec6d2d1aefd3","Type":"ContainerStarted","Data":"1608930d13e04d67fb94a5485554e2dac2a4f25c0bf4ab88995a7311e88d5a40"} Nov 02 23:03:57.951971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:57.951868 5672 generic.go:356] "Generic (PLEG): container finished" podID="3f23f148-51d0-48be-83eb-ec6d2d1aefd3" containerID="dbdeb2bc754fcc315b004c6b0f384d8c06ac10ed10d08b876f7bb7e072ebd613" exitCode=0 Nov 02 23:03:57.951971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:57.951950 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" event={"ID":"3f23f148-51d0-48be-83eb-ec6d2d1aefd3","Type":"ContainerDied","Data":"dbdeb2bc754fcc315b004c6b0f384d8c06ac10ed10d08b876f7bb7e072ebd613"} Nov 02 23:03:59.086704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.086677 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:59.213251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.213134 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-xks-ca\") pod \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " Nov 02 23:03:59.213251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.213221 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-openshift-ca\") pod \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " Nov 02 23:03:59.213497 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.213299 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-kube-api-access-gd6kj\" (UniqueName: \"kubernetes.io/projected/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-kube-api-access-gd6kj\") pod \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\" (UID: \"3f23f148-51d0-48be-83eb-ec6d2d1aefd3\") " Nov 02 23:03:59.213652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.213624 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-openshift-ca" pod "3f23f148-51d0-48be-83eb-ec6d2d1aefd3" (UID: "3f23f148-51d0-48be-83eb-ec6d2d1aefd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:03:59.215345 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.215317 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-xks-ca" pod "3f23f148-51d0-48be-83eb-ec6d2d1aefd3" (UID: "3f23f148-51d0-48be-83eb-ec6d2d1aefd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:03:59.215511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.215487 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-kube-api-access-gd6kj" pod "3f23f148-51d0-48be-83eb-ec6d2d1aefd3" (UID: "3f23f148-51d0-48be-83eb-ec6d2d1aefd3"). InnerVolumeSpecName "kube-api-access-gd6kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:03:59.314383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.314346 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:59.314383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.314379 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:59.314383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.314389 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd6kj\" (UniqueName: \"kubernetes.io/projected/3f23f148-51d0-48be-83eb-ec6d2d1aefd3-kube-api-access-gd6kj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:03:59.962970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.962939 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" Nov 02 23:03:59.963163 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.962964 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk" event={"ID":"3f23f148-51d0-48be-83eb-ec6d2d1aefd3","Type":"ContainerDied","Data":"1608930d13e04d67fb94a5485554e2dac2a4f25c0bf4ab88995a7311e88d5a40"} Nov 02 23:03:59.963163 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:03:59.963017 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1608930d13e04d67fb94a5485554e2dac2a4f25c0bf4ab88995a7311e88d5a40" Nov 02 23:04:00.133389 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:00.133356 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz"] Nov 02 23:04:00.135920 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:00.135885 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8a68c823-tfhqz"] Nov 02 23:04:00.711620 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:00.711584 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fef8f9-5a26-417d-8bce-03411fb862c6" path="/var/lib/kubelet/pods/f1fef8f9-5a26-417d-8bce-03411fb862c6/volumes" Nov 02 23:04:06.057992 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:06.057904 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb"] Nov 02 23:04:06.104354 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:06.104317 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0078993-kwjbb"] Nov 02 23:04:06.711402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:06.711373 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d086d657-6a6e-49a1-be0d-79f7322dd334" path="/var/lib/kubelet/pods/d086d657-6a6e-49a1-be0d-79f7322dd334/volumes" Nov 02 23:04:06.749359 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:06.749324 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:04:07.823482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.823449 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8"] Nov 02 23:04:07.823861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.823840 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3f23f148-51d0-48be-83eb-ec6d2d1aefd3" containerName="revoke-keys" Nov 02 23:04:07.823861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.823851 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f23f148-51d0-48be-83eb-ec6d2d1aefd3" containerName="revoke-keys" Nov 02 23:04:07.823952 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.823938 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f23f148-51d0-48be-83eb-ec6d2d1aefd3" containerName="revoke-keys" Nov 02 23:04:07.828467 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.828445 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:07.894476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.894435 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp5fs\" (UniqueName: \"kubernetes.io/projected/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-kube-api-access-zp5fs\") pod \"maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:07.894666 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.894600 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:07.894734 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.894715 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:07.918512 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.918475 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8"] Nov 02 23:04:07.926471 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.926440 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:04:07.926655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.926524 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:04:07.926655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.926579 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:04:07.926655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.926655 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:04:07.995831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.995794 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:07.996015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.995898 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:07.996015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.995931 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp5fs\" (UniqueName: \"kubernetes.io/projected/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-kube-api-access-zp5fs\") pod \"maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:07.996505 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.996483 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:07.998326 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:07.998301 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:08.056034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:08.055994 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp5fs\" (UniqueName: \"kubernetes.io/projected/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-kube-api-access-zp5fs\") pod \"maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:08.141979 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:08.141882 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:08.516481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:08.516267 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8"] Nov 02 23:04:08.518877 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:04:08.518847 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9e06ef_6cf8_44fa_a5ca_94b81ab9f884.slice/crio-37948f5eea069f8ec76afb4d2296961248997ca06d7504e43e5214ed23330515 WatchSource:0}: Error finding container 37948f5eea069f8ec76afb4d2296961248997ca06d7504e43e5214ed23330515: Status 404 returned error can't find the container with id 37948f5eea069f8ec76afb4d2296961248997ca06d7504e43e5214ed23330515 Nov 02 23:04:09.013207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:09.013171 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" event={"ID":"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884","Type":"ContainerStarted","Data":"83f412b30c0123b65cbc80990957b3ece996255ea6788cdca01929461efac7af"} Nov 02 23:04:09.013599 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:09.013252 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" event={"ID":"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884","Type":"ContainerStarted","Data":"37948f5eea069f8ec76afb4d2296961248997ca06d7504e43e5214ed23330515"} Nov 02 23:04:09.118890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:09.118776 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" podStartSLOduration=2.118759532 podStartE2EDuration="2.118759532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:09.117914296 +0000 UTC m=+642.525226137" watchObservedRunningTime="2026-11-02 23:04:09.118759532 +0000 UTC m=+642.526071365" Nov 02 23:04:10.019265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:10.019230 5672 generic.go:356] "Generic (PLEG): container finished" podID="bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884" containerID="83f412b30c0123b65cbc80990957b3ece996255ea6788cdca01929461efac7af" exitCode=0 Nov 02 23:04:10.019653 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:10.019313 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" event={"ID":"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884","Type":"ContainerDied","Data":"83f412b30c0123b65cbc80990957b3ece996255ea6788cdca01929461efac7af"} Nov 02 23:04:11.167377 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:11.167344 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:11.227881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:11.227835 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-openshift-ca\") pod \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " Nov 02 23:04:11.228053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:11.227959 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-kube-api-access-zp5fs\" (UniqueName: \"kubernetes.io/projected/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-kube-api-access-zp5fs\") pod \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " Nov 02 23:04:11.228053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:11.227996 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-xks-ca\") pod \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\" (UID: \"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884\") " Nov 02 23:04:11.228262 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:11.228240 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-openshift-ca" pod "bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884" (UID: "bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:04:11.228361 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:11.228341 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:11.230228 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:11.230201 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-xks-ca" pod "bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884" (UID: "bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:04:11.230333 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:11.230260 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-kube-api-access-zp5fs" pod "bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884" (UID: "bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884"). InnerVolumeSpecName "kube-api-access-zp5fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:04:11.329485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:11.329390 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:11.329485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:11.329433 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp5fs\" (UniqueName: \"kubernetes.io/projected/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884-kube-api-access-zp5fs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:04:12.034337 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:12.034303 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" Nov 02 23:04:12.034337 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:12.034314 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8" event={"ID":"bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884","Type":"ContainerDied","Data":"37948f5eea069f8ec76afb4d2296961248997ca06d7504e43e5214ed23330515"} Nov 02 23:04:12.034569 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:12.034378 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37948f5eea069f8ec76afb4d2296961248997ca06d7504e43e5214ed23330515" Nov 02 23:04:20.646886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:20.646846 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:04:26.020693 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:04:26.020651 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:04:36.419363 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:36.419327 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:04:37.004880 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:37.004845 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk"] Nov 02 23:04:37.053789 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:37.053752 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16aef2f3-5jhsk"] Nov 02 23:04:38.713569 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:38.713537 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f23f148-51d0-48be-83eb-ec6d2d1aefd3" path="/var/lib/kubelet/pods/3f23f148-51d0-48be-83eb-ec6d2d1aefd3/volumes" Nov 02 23:04:39.749101 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:39.749062 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8"] Nov 02 23:04:39.795494 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:39.795460 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db3e5cc4-5lvt8"] Nov 02 23:04:40.711217 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:40.711177 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884" path="/var/lib/kubelet/pods/bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884/volumes" Nov 02 23:04:54.759754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:04:54.759721 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:04:56.019599 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:04:56.019561 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:05:08.611039 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:08.611002 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:05:13.612090 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.612054 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z"] Nov 02 23:05:13.612503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.612436 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884" containerName="revoke-keys" Nov 02 23:05:13.612503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.612447 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884" containerName="revoke-keys" Nov 02 23:05:13.612573 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.612526 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9e06ef-6cf8-44fa-a5ca-94b81ab9f884" containerName="revoke-keys" Nov 02 23:05:13.615336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.615321 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.683864 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.683835 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-59f97a\"/\"kube-root-ca.crt\"" Nov 02 23:05:13.712470 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.712434 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-59f97a\"/\"openshift-service-ca.crt\"" Nov 02 23:05:13.712669 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.712540 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-59f97a\"/\"simulator-w0-59f97a-kserve-self-signed-certs\"" Nov 02 23:05:13.712766 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.712750 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-59f97a\"/\"default-dockercfg-lln4k\"" Nov 02 23:05:13.714367 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.714342 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z"] Nov 02 23:05:13.717185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.717166 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-model-cache\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.717287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.717266 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tls-certs\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.717346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.717335 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tmp-dir\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.717405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.717389 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-dshm\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.717442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.717414 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkn8g\" (UniqueName: \"kubernetes.io/projected/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-kube-api-access-lkn8g\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.717500 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.717485 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-home\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.818214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.818174 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tmp-dir\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.818214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.818214 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-dshm\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.818491 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.818231 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkn8g\" (UniqueName: \"kubernetes.io/projected/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-kube-api-access-lkn8g\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.818491 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.818290 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-home\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.818491 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.818315 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-model-cache\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.818707 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.818671 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tls-certs\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.818796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.818763 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-home\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.818856 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.818832 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-model-cache\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.818902 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.818880 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tmp-dir\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.820621 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.820599 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-dshm\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.820994 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.820976 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tls-certs\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.872449 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.872377 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkn8g\" (UniqueName: \"kubernetes.io/projected/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-kube-api-access-lkn8g\") pod \"simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:13.931553 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:13.931513 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:14.297171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:14.297127 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z"] Nov 02 23:05:14.299998 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:05:14.299971 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d9b5e4a_b530_4bf9_a6a0_c0bf99af8d78.slice/crio-f9a5b567fd8cd6fb7e329468d7ae99ec3897363ba103ece384a8c8f77b44c60a WatchSource:0}: Error finding container f9a5b567fd8cd6fb7e329468d7ae99ec3897363ba103ece384a8c8f77b44c60a: Status 404 returned error can't find the container with id f9a5b567fd8cd6fb7e329468d7ae99ec3897363ba103ece384a8c8f77b44c60a Nov 02 23:05:14.302060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:14.302041 5672 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:05:14.367928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:14.367893 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" event={"ID":"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78","Type":"ContainerStarted","Data":"f9a5b567fd8cd6fb7e329468d7ae99ec3897363ba103ece384a8c8f77b44c60a"} Nov 02 23:05:17.388748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:17.388710 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" event={"ID":"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78","Type":"ContainerStarted","Data":"cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d"} Nov 02 23:05:17.496439 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:17.496384 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podStartSLOduration=2.260160378 podStartE2EDuration="4.49636906s" totalImagesPullingTime="2.236208682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:17.49485733 +0000 UTC m=+710.902169159" watchObservedRunningTime="2026-11-02 23:05:17.49636906 +0000 UTC m=+710.903680910" Nov 02 23:05:17.932352 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:17.932308 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:19.045050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:19.045015 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:05:23.931808 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:23.931773 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:23.944310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:23.944285 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:24.440972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:24.440941 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:05:26.020701 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:05:26.020669 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:05:26.973482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:26.973450 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4"] Nov 02 23:05:26.977136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:26.977111 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.044636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.044599 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-home\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.045058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.044658 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-model-cache\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.045058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.044695 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-dshm\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.045058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.044733 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04740004-3f15-406b-a220-8926e141df5e-tls-certs\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.045058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.044781 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjnh\" (UniqueName: \"kubernetes.io/projected/04740004-3f15-406b-a220-8926e141df5e-kube-api-access-6kjnh\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.045058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.044973 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-tmp-dir\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.074292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.074256 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"openshift-service-ca.crt\"" Nov 02 23:05:27.074292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.074256 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"kube-root-ca.crt\"" Nov 02 23:05:27.074547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.074471 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"default-dockercfg-px9j7\"" Nov 02 23:05:27.074547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.074489 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"premium-w3-14b321-kserve-self-signed-certs\"" Nov 02 23:05:27.076070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.076049 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4"] Nov 02 23:05:27.145669 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.145636 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04740004-3f15-406b-a220-8926e141df5e-tls-certs\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.145854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.145680 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjnh\" (UniqueName: \"kubernetes.io/projected/04740004-3f15-406b-a220-8926e141df5e-kube-api-access-6kjnh\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.145854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.145711 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-tmp-dir\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.145854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.145750 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-home\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.145854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.145771 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-model-cache\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.145854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.145792 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-dshm\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.146243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.146222 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-tmp-dir\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.146341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.146248 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-model-cache\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.146341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.146286 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-home\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.148000 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.147973 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-dshm\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.148402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.148382 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04740004-3f15-406b-a220-8926e141df5e-tls-certs\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.198865 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.198831 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjnh\" (UniqueName: \"kubernetes.io/projected/04740004-3f15-406b-a220-8926e141df5e-kube-api-access-6kjnh\") pod \"premium-w3-14b321-kserve-bbccc9f74-z7bh4\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.288848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.288753 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:27.658367 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:27.658290 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4"] Nov 02 23:05:27.661166 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:05:27.661127 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04740004_3f15_406b_a220_8926e141df5e.slice/crio-b775b51d048c60eda954f59eae49962cddefb765d87ce3cdd405f9008eb2df1e WatchSource:0}: Error finding container b775b51d048c60eda954f59eae49962cddefb765d87ce3cdd405f9008eb2df1e: Status 404 returned error can't find the container with id b775b51d048c60eda954f59eae49962cddefb765d87ce3cdd405f9008eb2df1e Nov 02 23:05:28.446404 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:28.446372 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" event={"ID":"04740004-3f15-406b-a220-8926e141df5e","Type":"ContainerStarted","Data":"b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a"} Nov 02 23:05:28.446404 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:28.446410 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" event={"ID":"04740004-3f15-406b-a220-8926e141df5e","Type":"ContainerStarted","Data":"b775b51d048c60eda954f59eae49962cddefb765d87ce3cdd405f9008eb2df1e"} Nov 02 23:05:28.554530 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:28.554464 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podStartSLOduration=2.55444629 podStartE2EDuration="2.55444629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:28.552781573 +0000 UTC m=+721.960093400" watchObservedRunningTime="2026-11-02 23:05:28.55444629 +0000 UTC m=+721.961758122" Nov 02 23:05:29.289115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:29.289078 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:37.289496 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:37.289400 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:37.302215 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:37.302190 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:37.506826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:37.506795 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:05:41.909766 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:41.909732 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:05:47.312055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.312014 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq"] Nov 02 23:05:47.315697 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.315672 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.392462 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.392423 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq"] Nov 02 23:05:47.413760 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.413725 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ba1918\"/\"simulator-w2-ba1918-kserve-self-signed-certs\"" Nov 02 23:05:47.413927 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.413870 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ba1918\"/\"default-dockercfg-gpvcd\"" Nov 02 23:05:47.431381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.431342 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ba1918\"/\"openshift-service-ca.crt\"" Nov 02 23:05:47.431576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.431558 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ba1918\"/\"kube-root-ca.crt\"" Nov 02 23:05:47.438358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.438327 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-dshm\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.438545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.438415 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-model-cache\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.438545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.438443 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-home\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.438545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.438460 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tls-certs\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.438545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.438522 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tmp-dir\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.438697 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.438614 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42dpf\" (UniqueName: \"kubernetes.io/projected/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-kube-api-access-42dpf\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.539636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.539590 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-dshm\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.539841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.539706 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-model-cache\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.539841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.539746 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-home\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.539841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.539770 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tls-certs\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.539993 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.539840 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tmp-dir\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.540289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.540264 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-model-cache\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.540466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.540264 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-home\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.540466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.540453 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tmp-dir\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.540648 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.540279 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42dpf\" (UniqueName: \"kubernetes.io/projected/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-kube-api-access-42dpf\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.542906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.542882 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-dshm\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.543249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.543225 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tls-certs\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.593969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.593883 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42dpf\" (UniqueName: \"kubernetes.io/projected/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-kube-api-access-42dpf\") pod \"simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:47.627872 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:47.627836 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:48.005481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:48.005426 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq"] Nov 02 23:05:48.007745 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:05:48.007716 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod610e4cbd_8e1f_4af7_8ef0_f7ecf7b0162d.slice/crio-663c037ca9ad29fbd8444b0ccca84bae5a48b2d9b495a0ce909cd09956667c6c WatchSource:0}: Error finding container 663c037ca9ad29fbd8444b0ccca84bae5a48b2d9b495a0ce909cd09956667c6c: Status 404 returned error can't find the container with id 663c037ca9ad29fbd8444b0ccca84bae5a48b2d9b495a0ce909cd09956667c6c Nov 02 23:05:48.556332 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:48.556294 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" event={"ID":"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d","Type":"ContainerStarted","Data":"f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b"} Nov 02 23:05:48.556779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:48.556345 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" event={"ID":"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d","Type":"ContainerStarted","Data":"663c037ca9ad29fbd8444b0ccca84bae5a48b2d9b495a0ce909cd09956667c6c"} Nov 02 23:05:48.628120 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:48.628081 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:48.669807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:48.669728 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podStartSLOduration=1.669709643 podStartE2EDuration="1.669709643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:48.668287493 +0000 UTC m=+742.075599323" watchObservedRunningTime="2026-11-02 23:05:48.669709643 +0000 UTC m=+742.077021468" Nov 02 23:05:53.793792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.793761 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h"] Nov 02 23:05:53.797513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.797489 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:53.846354 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.846317 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-59f97a-openshift-default-dockercfg-r7nkq\"" Nov 02 23:05:53.894577 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.894534 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h"] Nov 02 23:05:53.901897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.901865 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:53.902079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.901914 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:53.902079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.902017 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:53.902079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.902046 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:53.902079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.902070 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:53.902347 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.902197 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplmp\" (UniqueName: \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-kube-api-access-pplmp\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:53.902347 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.902262 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:53.902347 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.902297 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:53.902512 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:53.902355 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.003674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.003627 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.003876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.003682 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.003876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.003738 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.003876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.003789 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.003876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.003817 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.003876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.003847 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.004130 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.003883 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplmp\" (UniqueName: \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-kube-api-access-pplmp\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.004130 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.004023 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.004130 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.004072 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.004332 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.004219 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.004397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.004329 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.004658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.004630 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.004774 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.004656 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.004903 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.004881 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.006505 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.006478 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.006598 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.006556 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.057196 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.057124 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.057595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.057576 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplmp\" (UniqueName: \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-kube-api-access-pplmp\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.110701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.110669 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:54.523299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.523273 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h"] Nov 02 23:05:54.525395 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:05:54.525361 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351d4c63_e1a7_4f82_a155_1c43ebdf3311.slice/crio-80b2ad27f9a47cca3c056c915166696de30567853133bb84b90bddd4b65916d1 WatchSource:0}: Error finding container 80b2ad27f9a47cca3c056c915166696de30567853133bb84b90bddd4b65916d1: Status 404 returned error can't find the container with id 80b2ad27f9a47cca3c056c915166696de30567853133bb84b90bddd4b65916d1 Nov 02 23:05:54.527999 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.527963 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:05:54.528091 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.528030 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:05:54.528091 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.528057 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:05:54.591584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.591549 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" event={"ID":"351d4c63-e1a7-4f82-a155-1c43ebdf3311","Type":"ContainerStarted","Data":"564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6"} Nov 02 23:05:54.591719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.591617 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" event={"ID":"351d4c63-e1a7-4f82-a155-1c43ebdf3311","Type":"ContainerStarted","Data":"80b2ad27f9a47cca3c056c915166696de30567853133bb84b90bddd4b65916d1"} Nov 02 23:05:54.702628 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:54.702576 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" podStartSLOduration=1.702560347 podStartE2EDuration="1.702560347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:54.700537007 +0000 UTC m=+748.107848837" watchObservedRunningTime="2026-11-02 23:05:54.702560347 +0000 UTC m=+748.109872171" Nov 02 23:05:55.111864 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:55.111828 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:05:55.113391 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:55.113364 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:05:55.113529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:55.113413 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" podUID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:05:56.020948 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:05:56.020883 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:05:56.111443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:56.111407 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:05:56.111614 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:56.111470 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" podUID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:05:57.111676 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:57.111646 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:05:57.112070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:57.111698 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" podUID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:05:57.643522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:57.643491 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:57.662050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:57.662018 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:58.111447 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:58.111411 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:05:58.111604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:58.111468 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" podUID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:05:58.637709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:58.637455 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:05:59.111830 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:59.111797 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:05:59.112019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:05:59.111856 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" podUID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:06:00.111503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:00.111466 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 23:06:00.111908 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:00.111522 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" podUID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 23:06:01.202434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:01.202342 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:06:01.202901 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:01.202774 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:06:01.204519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:01.204497 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:06:05.489504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:05.489460 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:06:13.227862 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.227829 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp"] Nov 02 23:06:13.234572 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.232559 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.308203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.308135 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-dshm\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.308420 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.308218 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjhp\" (UniqueName: \"kubernetes.io/projected/13f2410f-6859-45b8-8f8b-2397e063ba1e-kube-api-access-hgjhp\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.308420 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.308349 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-model-cache\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.308511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.308429 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f2410f-6859-45b8-8f8b-2397e063ba1e-tls-certs\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.308511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.308461 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-home\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.308511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.308488 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-tmp-dir\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.328622 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.328593 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-59f97a\"/\"premium-w0-59f97a-kserve-self-signed-certs\"" Nov 02 23:06:13.330527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.330505 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp"] Nov 02 23:06:13.409488 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.409455 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f2410f-6859-45b8-8f8b-2397e063ba1e-tls-certs\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.409488 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.409499 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-home\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.409722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.409530 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-tmp-dir\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.409722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.409646 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-dshm\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.409722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.409677 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjhp\" (UniqueName: \"kubernetes.io/projected/13f2410f-6859-45b8-8f8b-2397e063ba1e-kube-api-access-hgjhp\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.409722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.409705 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-model-cache\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.409960 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.409934 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-home\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.410030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.409983 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-tmp-dir\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.410117 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.410100 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-model-cache\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.411906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.411877 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-dshm\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.412031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.412016 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f2410f-6859-45b8-8f8b-2397e063ba1e-tls-certs\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.462971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.462934 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjhp\" (UniqueName: \"kubernetes.io/projected/13f2410f-6859-45b8-8f8b-2397e063ba1e-kube-api-access-hgjhp\") pod \"premium-w0-59f97a-kserve-7554cc7c55-4jbjp\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.550685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.550581 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:13.924604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:13.924541 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp"] Nov 02 23:06:13.926905 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:06:13.926875 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13f2410f_6859_45b8_8f8b_2397e063ba1e.slice/crio-562769ad925ffa675e5575b6fcc7a2fa7a59a377a167f5104aae7cd91b275c86 WatchSource:0}: Error finding container 562769ad925ffa675e5575b6fcc7a2fa7a59a377a167f5104aae7cd91b275c86: Status 404 returned error can't find the container with id 562769ad925ffa675e5575b6fcc7a2fa7a59a377a167f5104aae7cd91b275c86 Nov 02 23:06:14.722738 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:14.722701 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" event={"ID":"13f2410f-6859-45b8-8f8b-2397e063ba1e","Type":"ContainerStarted","Data":"d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62"} Nov 02 23:06:14.723172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:14.722751 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" event={"ID":"13f2410f-6859-45b8-8f8b-2397e063ba1e","Type":"ContainerStarted","Data":"562769ad925ffa675e5575b6fcc7a2fa7a59a377a167f5104aae7cd91b275c86"} Nov 02 23:06:14.829768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:14.829711 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podStartSLOduration=1.829694443 podStartE2EDuration="1.829694443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:14.829462299 +0000 UTC m=+768.236774152" watchObservedRunningTime="2026-11-02 23:06:14.829694443 +0000 UTC m=+768.237006291" Nov 02 23:06:15.551674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:15.551612 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:20.487040 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:20.487006 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:06:23.551310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:23.551280 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:23.563493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:23.563464 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:23.784273 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:23.784245 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:06:26.021187 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:06:26.021129 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:06:30.854521 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:30.854485 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt"] Nov 02 23:06:30.859365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:30.859339 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:30.911138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:30.911106 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt"] Nov 02 23:06:30.954952 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:30.954924 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:06:30.954952 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:30.954925 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:06:30.955222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:30.955125 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:06:30.955222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:30.955141 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:06:31.003571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.003538 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872d6677-hlkqt\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:31.003752 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.003584 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5x7j\" (UniqueName: \"kubernetes.io/projected/7e898c01-bb37-4fba-867b-e67b65862162-kube-api-access-x5x7j\") pod \"maas-api-revoke-sub-872d6677-hlkqt\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:31.003813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.003782 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872d6677-hlkqt\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:31.104805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.104702 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872d6677-hlkqt\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:31.104972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.104839 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872d6677-hlkqt\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:31.104972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.104869 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5x7j\" (UniqueName: \"kubernetes.io/projected/7e898c01-bb37-4fba-867b-e67b65862162-kube-api-access-x5x7j\") pod \"maas-api-revoke-sub-872d6677-hlkqt\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:31.105583 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.105555 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872d6677-hlkqt\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:31.107282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.107257 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872d6677-hlkqt\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:31.158387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.158351 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5x7j\" (UniqueName: \"kubernetes.io/projected/7e898c01-bb37-4fba-867b-e67b65862162-kube-api-access-x5x7j\") pod \"maas-api-revoke-sub-872d6677-hlkqt\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:31.172251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.172220 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:31.541718 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.541595 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt"] Nov 02 23:06:31.544037 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:06:31.544009 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e898c01_bb37_4fba_867b_e67b65862162.slice/crio-25bf2b9893e88eb750c6b20507080723d776b9bf51facedbffee6358edb98abc WatchSource:0}: Error finding container 25bf2b9893e88eb750c6b20507080723d776b9bf51facedbffee6358edb98abc: Status 404 returned error can't find the container with id 25bf2b9893e88eb750c6b20507080723d776b9bf51facedbffee6358edb98abc Nov 02 23:06:31.821108 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:31.821028 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" event={"ID":"7e898c01-bb37-4fba-867b-e67b65862162","Type":"ContainerStarted","Data":"25bf2b9893e88eb750c6b20507080723d776b9bf51facedbffee6358edb98abc"} Nov 02 23:06:32.827817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:32.827778 5672 generic.go:356] "Generic (PLEG): container finished" podID="7e898c01-bb37-4fba-867b-e67b65862162" containerID="db90600607de556857fd500bab8397037d6837e4c1f7d118333525346dc4d0cb" exitCode=0 Nov 02 23:06:32.828247 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:32.827869 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" event={"ID":"7e898c01-bb37-4fba-867b-e67b65862162","Type":"ContainerDied","Data":"db90600607de556857fd500bab8397037d6837e4c1f7d118333525346dc4d0cb"} Nov 02 23:06:33.974340 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:33.974317 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:34.136433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.136328 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e898c01-bb37-4fba-867b-e67b65862162-kube-api-access-x5x7j\" (UniqueName: \"kubernetes.io/projected/7e898c01-bb37-4fba-867b-e67b65862162-kube-api-access-x5x7j\") pod \"7e898c01-bb37-4fba-867b-e67b65862162\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " Nov 02 23:06:34.136433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.136417 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-xks-ca\") pod \"7e898c01-bb37-4fba-867b-e67b65862162\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " Nov 02 23:06:34.136657 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.136498 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-openshift-ca\") pod \"7e898c01-bb37-4fba-867b-e67b65862162\" (UID: \"7e898c01-bb37-4fba-867b-e67b65862162\") " Nov 02 23:06:34.136890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.136850 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-openshift-ca" pod "7e898c01-bb37-4fba-867b-e67b65862162" (UID: "7e898c01-bb37-4fba-867b-e67b65862162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:06:34.138801 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.138761 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-xks-ca" pod "7e898c01-bb37-4fba-867b-e67b65862162" (UID: "7e898c01-bb37-4fba-867b-e67b65862162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:06:34.138928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.138864 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e898c01-bb37-4fba-867b-e67b65862162-kube-api-access-x5x7j" pod "7e898c01-bb37-4fba-867b-e67b65862162" (UID: "7e898c01-bb37-4fba-867b-e67b65862162"). InnerVolumeSpecName "kube-api-access-x5x7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:06:34.237708 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.237668 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5x7j\" (UniqueName: \"kubernetes.io/projected/7e898c01-bb37-4fba-867b-e67b65862162-kube-api-access-x5x7j\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:06:34.237708 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.237703 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:06:34.237909 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.237718 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e898c01-bb37-4fba-867b-e67b65862162-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:06:34.842036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.842003 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" Nov 02 23:06:34.842036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.842016 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt" event={"ID":"7e898c01-bb37-4fba-867b-e67b65862162","Type":"ContainerDied","Data":"25bf2b9893e88eb750c6b20507080723d776b9bf51facedbffee6358edb98abc"} Nov 02 23:06:34.842317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:34.842059 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25bf2b9893e88eb750c6b20507080723d776b9bf51facedbffee6358edb98abc" Nov 02 23:06:38.817897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:38.817856 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:06:48.650373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.650333 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc"] Nov 02 23:06:48.650938 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.650921 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7e898c01-bb37-4fba-867b-e67b65862162" containerName="revoke-keys" Nov 02 23:06:48.650985 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.650940 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e898c01-bb37-4fba-867b-e67b65862162" containerName="revoke-keys" Nov 02 23:06:48.651077 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.651066 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e898c01-bb37-4fba-867b-e67b65862162" containerName="revoke-keys" Nov 02 23:06:48.654695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.654675 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:48.673483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.673448 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e3ef70-4fqdc\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:48.673636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.673502 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czs98\" (UniqueName: \"kubernetes.io/projected/a2441016-6522-4f36-996e-e54ca4d73534-kube-api-access-czs98\") pod \"maas-api-revoke-sub-c4e3ef70-4fqdc\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:48.673737 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.673716 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e3ef70-4fqdc\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:48.750687 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.750643 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:06:48.750881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.750732 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:06:48.750881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.750741 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:06:48.750881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.750739 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:06:48.752564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.752538 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc"] Nov 02 23:06:48.774631 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.774607 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e3ef70-4fqdc\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:48.774785 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.774645 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e3ef70-4fqdc\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:48.774785 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.774685 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czs98\" (UniqueName: \"kubernetes.io/projected/a2441016-6522-4f36-996e-e54ca4d73534-kube-api-access-czs98\") pod \"maas-api-revoke-sub-c4e3ef70-4fqdc\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:48.775508 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.775476 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e3ef70-4fqdc\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:48.777365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.777341 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e3ef70-4fqdc\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:48.828324 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.828294 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czs98\" (UniqueName: \"kubernetes.io/projected/a2441016-6522-4f36-996e-e54ca4d73534-kube-api-access-czs98\") pod \"maas-api-revoke-sub-c4e3ef70-4fqdc\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:48.970174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:48.970124 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:49.330494 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:49.330469 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc"] Nov 02 23:06:49.333003 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:06:49.332971 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2441016_6522_4f36_996e_e54ca4d73534.slice/crio-5bcf3a5429f3d346a96f02b28ad583adc7beb70f2d2dd7b9e33fe3f862e4edef WatchSource:0}: Error finding container 5bcf3a5429f3d346a96f02b28ad583adc7beb70f2d2dd7b9e33fe3f862e4edef: Status 404 returned error can't find the container with id 5bcf3a5429f3d346a96f02b28ad583adc7beb70f2d2dd7b9e33fe3f862e4edef Nov 02 23:06:49.925667 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:49.925619 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" event={"ID":"a2441016-6522-4f36-996e-e54ca4d73534","Type":"ContainerStarted","Data":"3d40acf99988a7d3892891ad7b447679d9ac4358c8df7f650fc174a7d15dfaa0"} Nov 02 23:06:49.926098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:49.925702 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" event={"ID":"a2441016-6522-4f36-996e-e54ca4d73534","Type":"ContainerStarted","Data":"5bcf3a5429f3d346a96f02b28ad583adc7beb70f2d2dd7b9e33fe3f862e4edef"} Nov 02 23:06:50.036211 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:50.036138 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" podStartSLOduration=2.03612249 podStartE2EDuration="2.03612249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:50.034742815 +0000 UTC m=+803.442054647" watchObservedRunningTime="2026-11-02 23:06:50.03612249 +0000 UTC m=+803.443434313" Nov 02 23:06:50.932569 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:50.932535 5672 generic.go:356] "Generic (PLEG): container finished" podID="a2441016-6522-4f36-996e-e54ca4d73534" containerID="3d40acf99988a7d3892891ad7b447679d9ac4358c8df7f650fc174a7d15dfaa0" exitCode=0 Nov 02 23:06:50.932949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:50.932628 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" event={"ID":"a2441016-6522-4f36-996e-e54ca4d73534","Type":"ContainerDied","Data":"3d40acf99988a7d3892891ad7b447679d9ac4358c8df7f650fc174a7d15dfaa0"} Nov 02 23:06:52.074768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.074744 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:52.110011 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.109981 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2441016-6522-4f36-996e-e54ca4d73534-kube-api-access-czs98\" (UniqueName: \"kubernetes.io/projected/a2441016-6522-4f36-996e-e54ca4d73534-kube-api-access-czs98\") pod \"a2441016-6522-4f36-996e-e54ca4d73534\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " Nov 02 23:06:52.110171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.110043 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-xks-ca\") pod \"a2441016-6522-4f36-996e-e54ca4d73534\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " Nov 02 23:06:52.110171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.110074 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-openshift-ca\") pod \"a2441016-6522-4f36-996e-e54ca4d73534\" (UID: \"a2441016-6522-4f36-996e-e54ca4d73534\") " Nov 02 23:06:52.110484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.110447 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-openshift-ca" pod "a2441016-6522-4f36-996e-e54ca4d73534" (UID: "a2441016-6522-4f36-996e-e54ca4d73534"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:06:52.112200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.112172 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-xks-ca" pod "a2441016-6522-4f36-996e-e54ca4d73534" (UID: "a2441016-6522-4f36-996e-e54ca4d73534"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:06:52.112296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.112217 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2441016-6522-4f36-996e-e54ca4d73534-kube-api-access-czs98" pod "a2441016-6522-4f36-996e-e54ca4d73534" (UID: "a2441016-6522-4f36-996e-e54ca4d73534"). InnerVolumeSpecName "kube-api-access-czs98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:06:52.211834 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.211747 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:06:52.211834 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.211776 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2441016-6522-4f36-996e-e54ca4d73534-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:06:52.211834 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.211786 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czs98\" (UniqueName: \"kubernetes.io/projected/a2441016-6522-4f36-996e-e54ca4d73534-kube-api-access-czs98\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:06:52.946374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.946333 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" event={"ID":"a2441016-6522-4f36-996e-e54ca4d73534","Type":"ContainerDied","Data":"5bcf3a5429f3d346a96f02b28ad583adc7beb70f2d2dd7b9e33fe3f862e4edef"} Nov 02 23:06:52.946374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.946374 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc" Nov 02 23:06:52.946594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:06:52.946396 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bcf3a5429f3d346a96f02b28ad583adc7beb70f2d2dd7b9e33fe3f862e4edef" Nov 02 23:06:56.020597 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:06:56.020551 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:07:03.793325 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:03.793288 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:07:07.611641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.611561 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q"] Nov 02 23:07:07.612101 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.611964 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a2441016-6522-4f36-996e-e54ca4d73534" containerName="revoke-keys" Nov 02 23:07:07.612101 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.611975 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2441016-6522-4f36-996e-e54ca4d73534" containerName="revoke-keys" Nov 02 23:07:07.612246 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.612105 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2441016-6522-4f36-996e-e54ca4d73534" containerName="revoke-keys" Nov 02 23:07:07.616597 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.616573 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.668055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.667938 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3d8404\"/\"openshift-service-ca.crt\"" Nov 02 23:07:07.712301 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.712273 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3d8404\"/\"default-dockercfg-8s6dn\"" Nov 02 23:07:07.712301 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.712276 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3d8404\"/\"premium-w1-3d8404-kserve-self-signed-certs\"" Nov 02 23:07:07.712486 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.712310 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3d8404\"/\"kube-root-ca.crt\"" Nov 02 23:07:07.714241 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.714221 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q"] Nov 02 23:07:07.768761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.768726 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-dshm\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.768945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.768768 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6dgj\" (UniqueName: \"kubernetes.io/projected/1fe36396-633b-4974-9fdb-16cf3b7f62c7-kube-api-access-z6dgj\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.768945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.768899 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-home\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.768945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.768928 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tls-certs\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.769110 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.769037 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-model-cache\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.769110 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.769059 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tmp-dir\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.870036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.869948 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-model-cache\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.870036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.869988 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tmp-dir\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.870317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.870185 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-dshm\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.870317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.870246 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6dgj\" (UniqueName: \"kubernetes.io/projected/1fe36396-633b-4974-9fdb-16cf3b7f62c7-kube-api-access-z6dgj\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.870487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.870461 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-home\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.870679 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.870660 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tls-certs\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.870936 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.870587 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tmp-dir\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.870936 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.870548 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-model-cache\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.871332 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.871303 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-home\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.872854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.872834 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-dshm\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.873263 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.873247 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tls-certs\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.923862 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.923830 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6dgj\" (UniqueName: \"kubernetes.io/projected/1fe36396-633b-4974-9fdb-16cf3b7f62c7-kube-api-access-z6dgj\") pod \"premium-w1-3d8404-kserve-75f544c868-fch6q\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:07.958224 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:07.958143 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:08.341890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:08.341865 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q"] Nov 02 23:07:08.344020 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:07:08.343992 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe36396_633b_4974_9fdb_16cf3b7f62c7.slice/crio-84c736a67a5809ba013af01f00ee78a99b3555f56ce2ca34f0323b102077f949 WatchSource:0}: Error finding container 84c736a67a5809ba013af01f00ee78a99b3555f56ce2ca34f0323b102077f949: Status 404 returned error can't find the container with id 84c736a67a5809ba013af01f00ee78a99b3555f56ce2ca34f0323b102077f949 Nov 02 23:07:09.049160 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:09.049116 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" event={"ID":"1fe36396-633b-4974-9fdb-16cf3b7f62c7","Type":"ContainerStarted","Data":"cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993"} Nov 02 23:07:09.049550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:09.049172 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" event={"ID":"1fe36396-633b-4974-9fdb-16cf3b7f62c7","Type":"ContainerStarted","Data":"84c736a67a5809ba013af01f00ee78a99b3555f56ce2ca34f0323b102077f949"} Nov 02 23:07:09.155303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:09.155232 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podStartSLOduration=2.15521794 podStartE2EDuration="2.15521794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:07:09.154297919 +0000 UTC m=+822.561609746" watchObservedRunningTime="2026-11-02 23:07:09.15521794 +0000 UTC m=+822.562529769" Nov 02 23:07:09.959310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:09.959271 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:14.856321 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:14.856281 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6"] Nov 02 23:07:14.856823 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:14.856492 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" containerID="cri-o://457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7" gracePeriod=30 Nov 02 23:07:15.034928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.034894 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp"] Nov 02 23:07:15.043822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.043799 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:15.099053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.099009 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp"] Nov 02 23:07:15.102840 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.102810 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:07:15.103004 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.102821 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:07:15.103004 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.102847 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:07:15.103004 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.102920 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:07:15.143526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.143444 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3acc52c0-gmqnp\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:15.143660 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.143567 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jb6z\" (UniqueName: \"kubernetes.io/projected/f387b94a-653b-4756-ba6a-c599406b1abf-kube-api-access-7jb6z\") pod \"maas-api-revoke-sub-3acc52c0-gmqnp\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:15.143660 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.143620 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3acc52c0-gmqnp\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:15.244179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.244104 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3acc52c0-gmqnp\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:15.244385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.244225 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jb6z\" (UniqueName: \"kubernetes.io/projected/f387b94a-653b-4756-ba6a-c599406b1abf-kube-api-access-7jb6z\") pod \"maas-api-revoke-sub-3acc52c0-gmqnp\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:15.244385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.244286 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3acc52c0-gmqnp\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:15.245128 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.245102 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3acc52c0-gmqnp\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:15.246376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.246356 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3acc52c0-gmqnp\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:15.297278 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.297248 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jb6z\" (UniqueName: \"kubernetes.io/projected/f387b94a-653b-4756-ba6a-c599406b1abf-kube-api-access-7jb6z\") pod \"maas-api-revoke-sub-3acc52c0-gmqnp\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:15.356276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.356242 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:15.728135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:15.728108 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp"] Nov 02 23:07:15.730771 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:07:15.730739 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf387b94a_653b_4756_ba6a_c599406b1abf.slice/crio-2bd3e7a2690b745c0371820f4a16abd4bf049f7165bf5e6fd254187b5581a45f WatchSource:0}: Error finding container 2bd3e7a2690b745c0371820f4a16abd4bf049f7165bf5e6fd254187b5581a45f: Status 404 returned error can't find the container with id 2bd3e7a2690b745c0371820f4a16abd4bf049f7165bf5e6fd254187b5581a45f Nov 02 23:07:16.089814 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.089773 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" event={"ID":"f387b94a-653b-4756-ba6a-c599406b1abf","Type":"ContainerStarted","Data":"955bf124f9ccac4a82a3d6b16f4878bb8a21217f9e3e934734f08e394a9c0faa"} Nov 02 23:07:16.090247 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.089846 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" event={"ID":"f387b94a-653b-4756-ba6a-c599406b1abf","Type":"ContainerStarted","Data":"2bd3e7a2690b745c0371820f4a16abd4bf049f7165bf5e6fd254187b5581a45f"} Nov 02 23:07:16.196362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.196299 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" podStartSLOduration=2.19628509 podStartE2EDuration="2.19628509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:07:16.194489769 +0000 UTC m=+829.601801595" watchObservedRunningTime="2026-11-02 23:07:16.19628509 +0000 UTC m=+829.603596921" Nov 02 23:07:16.296018 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.295927 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k"] Nov 02 23:07:16.299759 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.299736 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:16.352098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.352063 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k"] Nov 02 23:07:16.354856 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.354825 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8781ed61-qfv9k\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:16.355002 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.354873 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8781ed61-qfv9k\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:16.355002 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.354906 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npsxb\" (UniqueName: \"kubernetes.io/projected/1d965abc-e8a2-4476-b130-9091c8197a15-kube-api-access-npsxb\") pod \"maas-api-revoke-sub-8781ed61-qfv9k\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:16.455541 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.455507 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8781ed61-qfv9k\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:16.455730 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.455559 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8781ed61-qfv9k\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:16.455730 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.455599 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsxb\" (UniqueName: \"kubernetes.io/projected/1d965abc-e8a2-4476-b130-9091c8197a15-kube-api-access-npsxb\") pod \"maas-api-revoke-sub-8781ed61-qfv9k\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:16.456462 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.456435 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8781ed61-qfv9k\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:16.457851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.457829 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8781ed61-qfv9k\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:16.509660 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.509631 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npsxb\" (UniqueName: \"kubernetes.io/projected/1d965abc-e8a2-4476-b130-9091c8197a15-kube-api-access-npsxb\") pod \"maas-api-revoke-sub-8781ed61-qfv9k\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:16.613493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.613418 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:16.982730 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:16.982704 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k"] Nov 02 23:07:16.984526 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:07:16.984501 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d965abc_e8a2_4476_b130_9091c8197a15.slice/crio-40a5a2dbb147a19e31115e7aedaaa60591904c422e8ea60304536569ba40f184 WatchSource:0}: Error finding container 40a5a2dbb147a19e31115e7aedaaa60591904c422e8ea60304536569ba40f184: Status 404 returned error can't find the container with id 40a5a2dbb147a19e31115e7aedaaa60591904c422e8ea60304536569ba40f184 Nov 02 23:07:17.096789 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:17.096750 5672 generic.go:356] "Generic (PLEG): container finished" podID="f387b94a-653b-4756-ba6a-c599406b1abf" containerID="955bf124f9ccac4a82a3d6b16f4878bb8a21217f9e3e934734f08e394a9c0faa" exitCode=0 Nov 02 23:07:17.097236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:17.096837 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" event={"ID":"f387b94a-653b-4756-ba6a-c599406b1abf","Type":"ContainerDied","Data":"955bf124f9ccac4a82a3d6b16f4878bb8a21217f9e3e934734f08e394a9c0faa"} Nov 02 23:07:17.098085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:17.098066 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" event={"ID":"1d965abc-e8a2-4476-b130-9091c8197a15","Type":"ContainerStarted","Data":"40a5a2dbb147a19e31115e7aedaaa60591904c422e8ea60304536569ba40f184"} Nov 02 23:07:17.959221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:17.959182 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:17.972674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:17.972644 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:18.105410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.105320 5672 generic.go:356] "Generic (PLEG): container finished" podID="1d965abc-e8a2-4476-b130-9091c8197a15" containerID="fc1a8c3e5118bd512d43154ecbe57c5b2718b555fa580472eb3ae3c798d556d0" exitCode=0 Nov 02 23:07:18.105862 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.105409 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" event={"ID":"1d965abc-e8a2-4476-b130-9091c8197a15","Type":"ContainerDied","Data":"fc1a8c3e5118bd512d43154ecbe57c5b2718b555fa580472eb3ae3c798d556d0"} Nov 02 23:07:18.117185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.117142 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:07:18.241948 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.241923 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:18.275111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.275077 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f387b94a-653b-4756-ba6a-c599406b1abf-kube-api-access-7jb6z\" (UniqueName: \"kubernetes.io/projected/f387b94a-653b-4756-ba6a-c599406b1abf-kube-api-access-7jb6z\") pod \"f387b94a-653b-4756-ba6a-c599406b1abf\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " Nov 02 23:07:18.275304 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.275206 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-openshift-ca\") pod \"f387b94a-653b-4756-ba6a-c599406b1abf\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " Nov 02 23:07:18.275358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.275313 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-xks-ca\") pod \"f387b94a-653b-4756-ba6a-c599406b1abf\" (UID: \"f387b94a-653b-4756-ba6a-c599406b1abf\") " Nov 02 23:07:18.275700 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.275570 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-openshift-ca" pod "f387b94a-653b-4756-ba6a-c599406b1abf" (UID: "f387b94a-653b-4756-ba6a-c599406b1abf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:07:18.275700 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.275658 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:18.277342 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.277314 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f387b94a-653b-4756-ba6a-c599406b1abf-kube-api-access-7jb6z" pod "f387b94a-653b-4756-ba6a-c599406b1abf" (UID: "f387b94a-653b-4756-ba6a-c599406b1abf"). InnerVolumeSpecName "kube-api-access-7jb6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:07:18.277450 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.277414 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-xks-ca" pod "f387b94a-653b-4756-ba6a-c599406b1abf" (UID: "f387b94a-653b-4756-ba6a-c599406b1abf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:07:18.376578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.376500 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f387b94a-653b-4756-ba6a-c599406b1abf-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:18.376578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:18.376530 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jb6z\" (UniqueName: \"kubernetes.io/projected/f387b94a-653b-4756-ba6a-c599406b1abf-kube-api-access-7jb6z\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:19.112527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.112440 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" Nov 02 23:07:19.112527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.112476 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp" event={"ID":"f387b94a-653b-4756-ba6a-c599406b1abf","Type":"ContainerDied","Data":"2bd3e7a2690b745c0371820f4a16abd4bf049f7165bf5e6fd254187b5581a45f"} Nov 02 23:07:19.112527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.112522 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd3e7a2690b745c0371820f4a16abd4bf049f7165bf5e6fd254187b5581a45f" Nov 02 23:07:19.253931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.253901 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:19.285282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.285246 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d965abc-e8a2-4476-b130-9091c8197a15-kube-api-access-npsxb\" (UniqueName: \"kubernetes.io/projected/1d965abc-e8a2-4476-b130-9091c8197a15-kube-api-access-npsxb\") pod \"1d965abc-e8a2-4476-b130-9091c8197a15\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " Nov 02 23:07:19.285495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.285305 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-openshift-ca\") pod \"1d965abc-e8a2-4476-b130-9091c8197a15\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " Nov 02 23:07:19.285495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.285353 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-xks-ca\") pod \"1d965abc-e8a2-4476-b130-9091c8197a15\" (UID: \"1d965abc-e8a2-4476-b130-9091c8197a15\") " Nov 02 23:07:19.285915 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.285879 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-openshift-ca" pod "1d965abc-e8a2-4476-b130-9091c8197a15" (UID: "1d965abc-e8a2-4476-b130-9091c8197a15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:07:19.288221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.288190 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d965abc-e8a2-4476-b130-9091c8197a15-kube-api-access-npsxb" pod "1d965abc-e8a2-4476-b130-9091c8197a15" (UID: "1d965abc-e8a2-4476-b130-9091c8197a15"). InnerVolumeSpecName "kube-api-access-npsxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:07:19.288336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.288294 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-xks-ca" pod "1d965abc-e8a2-4476-b130-9091c8197a15" (UID: "1d965abc-e8a2-4476-b130-9091c8197a15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:07:19.387124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.387029 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npsxb\" (UniqueName: \"kubernetes.io/projected/1d965abc-e8a2-4476-b130-9091c8197a15-kube-api-access-npsxb\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:19.387124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.387071 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:19.387124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.387089 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d965abc-e8a2-4476-b130-9091c8197a15-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:19.897991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.897965 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7jln6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 02 23:07:19.898099 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:19.898018 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 02 23:07:20.117091 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.117064 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 23:07:20.119609 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.119584 5672 generic.go:356] "Generic (PLEG): container finished" podID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerID="457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7" exitCode=0 Nov 02 23:07:20.119756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.119629 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" Nov 02 23:07:20.119756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.119667 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" event={"ID":"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b","Type":"ContainerDied","Data":"457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7"} Nov 02 23:07:20.119756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.119729 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6" event={"ID":"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b","Type":"ContainerDied","Data":"371a3ca8c9c1f56856180ffbad183337113b1c8b8b13a9939e491d839c9d9ee5"} Nov 02 23:07:20.119898 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.119757 5672 scope.go:122] "RemoveContainer" containerID="457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7" Nov 02 23:07:20.121094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.121077 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" Nov 02 23:07:20.121206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.121124 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k" event={"ID":"1d965abc-e8a2-4476-b130-9091c8197a15","Type":"ContainerDied","Data":"40a5a2dbb147a19e31115e7aedaaa60591904c422e8ea60304536569ba40f184"} Nov 02 23:07:20.121206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.121183 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40a5a2dbb147a19e31115e7aedaaa60591904c422e8ea60304536569ba40f184" Nov 02 23:07:20.130993 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.130977 5672 scope.go:122] "RemoveContainer" containerID="3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727" Nov 02 23:07:20.139289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.139270 5672 scope.go:122] "RemoveContainer" containerID="457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7" Nov 02 23:07:20.139599 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:07:20.139575 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7\": container with ID starting with 457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7 not found: ID does not exist" containerID="457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7" Nov 02 23:07:20.139688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.139607 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7"} err="failed to get container status \"457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7\": rpc error: code = NotFound desc = could not find container \"457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7\": container with ID starting with 457d56bca73de82073725d4667e3e8230f65b35894e5537dd1d10470b0461ef7 not found: ID does not exist" Nov 02 23:07:20.139688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.139625 5672 scope.go:122] "RemoveContainer" containerID="3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727" Nov 02 23:07:20.139898 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:07:20.139880 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727\": container with ID starting with 3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727 not found: ID does not exist" containerID="3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727" Nov 02 23:07:20.139957 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.139902 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727"} err="failed to get container status \"3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727\": rpc error: code = NotFound desc = could not find container \"3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727\": container with ID starting with 3fb69607cac1f64ef5dc89e6dccc95d6db57923dc24832ed662587b06c6a5727 not found: ID does not exist" Nov 02 23:07:20.195715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.195679 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-certs\") pod \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " Nov 02 23:07:20.195890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.195737 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-socket\") pod \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " Nov 02 23:07:20.195890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.195791 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-kube-api-access-nm447\" (UniqueName: \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-kube-api-access-nm447\") pod \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " Nov 02 23:07:20.195890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.195859 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-data\") pod \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " Nov 02 23:07:20.196073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.195898 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-token\" (UniqueName: \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-token\") pod \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " Nov 02 23:07:20.196073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.195941 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istiod-ca-cert\") pod \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " Nov 02 23:07:20.196073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.195968 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-podinfo\") pod \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " Nov 02 23:07:20.196073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.196006 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-credential-socket\") pod \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " Nov 02 23:07:20.196073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.196002 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-certs" pod "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" (UID: "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:07:20.196073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.196035 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-envoy\") pod \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\" (UID: \"7cd55d7c-49eb-40d2-9daf-49dd2996bc6b\") " Nov 02 23:07:20.196378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.196087 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-socket" pod "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" (UID: "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:07:20.196378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.196309 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-data" pod "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" (UID: "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:07:20.196378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.196317 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-credential-socket" pod "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" (UID: "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:07:20.196517 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.196418 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istiod-ca-cert" pod "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" (UID: "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:07:20.196517 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.196451 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:20.196517 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.196464 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:20.196517 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.196473 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:20.196517 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.196482 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:20.198599 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.198566 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-token" pod "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" (UID: "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:07:20.198599 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.198575 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-podinfo" pod "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" (UID: "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:07:20.198751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.198741 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-kube-api-access-nm447" pod "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" (UID: "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b"). InnerVolumeSpecName "kube-api-access-nm447". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:07:20.198824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.198750 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-envoy" pod "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" (UID: "7cd55d7c-49eb-40d2-9daf-49dd2996bc6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:07:20.297898 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.297870 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:20.297898 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.297897 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:20.298050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.297906 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:20.298050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.297916 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:20.298050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.297925 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm447\" (UniqueName: \"kubernetes.io/projected/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b-kube-api-access-nm447\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:20.537207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.537169 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6"] Nov 02 23:07:20.541338 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.541307 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7jln6"] Nov 02 23:07:20.711680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:20.711601 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" path="/var/lib/kubelet/pods/7cd55d7c-49eb-40d2-9daf-49dd2996bc6b/volumes" Nov 02 23:07:26.020164 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:07:26.020107 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:07:30.324712 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.324672 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2"] Nov 02 23:07:30.325185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.325048 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f387b94a-653b-4756-ba6a-c599406b1abf" containerName="revoke-keys" Nov 02 23:07:30.325185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.325060 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f387b94a-653b-4756-ba6a-c599406b1abf" containerName="revoke-keys" Nov 02 23:07:30.325185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.325072 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" Nov 02 23:07:30.325185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.325077 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" Nov 02 23:07:30.325185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.325106 5672 cpu_manager.go:414] "containerMap: removing container" podUID="1d965abc-e8a2-4476-b130-9091c8197a15" containerName="revoke-keys" Nov 02 23:07:30.325185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.325111 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d965abc-e8a2-4476-b130-9091c8197a15" containerName="revoke-keys" Nov 02 23:07:30.325185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.325188 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d965abc-e8a2-4476-b130-9091c8197a15" containerName="revoke-keys" Nov 02 23:07:30.325536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.325198 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f387b94a-653b-4756-ba6a-c599406b1abf" containerName="revoke-keys" Nov 02 23:07:30.325536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.325205 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" Nov 02 23:07:30.325536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.325213 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" Nov 02 23:07:30.329947 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.329924 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:30.382036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.381999 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2"] Nov 02 23:07:30.397229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.397199 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd6810ea-22sx2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:30.397388 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.397268 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfb8\" (UniqueName: \"kubernetes.io/projected/9c1929b2-4756-4bbe-986c-12d83d83d3c2-kube-api-access-slfb8\") pod \"maas-api-revoke-sub-dd6810ea-22sx2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:30.397476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.397453 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd6810ea-22sx2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:30.425466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.425434 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:07:30.425466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.425461 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:07:30.425654 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.425483 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:07:30.425654 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.425486 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:07:30.498636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.498588 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd6810ea-22sx2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:30.498830 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.498664 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfb8\" (UniqueName: \"kubernetes.io/projected/9c1929b2-4756-4bbe-986c-12d83d83d3c2-kube-api-access-slfb8\") pod \"maas-api-revoke-sub-dd6810ea-22sx2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:30.498830 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.498776 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd6810ea-22sx2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:30.499372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.499351 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd6810ea-22sx2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:30.501006 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.500987 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd6810ea-22sx2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:30.552541 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.552512 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfb8\" (UniqueName: \"kubernetes.io/projected/9c1929b2-4756-4bbe-986c-12d83d83d3c2-kube-api-access-slfb8\") pod \"maas-api-revoke-sub-dd6810ea-22sx2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:30.643245 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:30.643172 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:31.008348 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:31.008324 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2"] Nov 02 23:07:31.010594 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:07:31.010566 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1929b2_4756_4bbe_986c_12d83d83d3c2.slice/crio-c795a8f7d9593c6cb5e82ae822ba97d1a3225b48848ed5ff985c665be7db84ba WatchSource:0}: Error finding container c795a8f7d9593c6cb5e82ae822ba97d1a3225b48848ed5ff985c665be7db84ba: Status 404 returned error can't find the container with id c795a8f7d9593c6cb5e82ae822ba97d1a3225b48848ed5ff985c665be7db84ba Nov 02 23:07:31.188347 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:31.188313 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" event={"ID":"9c1929b2-4756-4bbe-986c-12d83d83d3c2","Type":"ContainerStarted","Data":"c795a8f7d9593c6cb5e82ae822ba97d1a3225b48848ed5ff985c665be7db84ba"} Nov 02 23:07:32.195099 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:32.195013 5672 generic.go:356] "Generic (PLEG): container finished" podID="9c1929b2-4756-4bbe-986c-12d83d83d3c2" containerID="8209a8616fa18ea1b0978f32270c00ccb7aa36eb2c4b5138c088fb6fd82a9779" exitCode=0 Nov 02 23:07:32.195553 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:32.195105 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" event={"ID":"9c1929b2-4756-4bbe-986c-12d83d83d3c2","Type":"ContainerDied","Data":"8209a8616fa18ea1b0978f32270c00ccb7aa36eb2c4b5138c088fb6fd82a9779"} Nov 02 23:07:33.341580 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:33.341552 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:33.428892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:33.428858 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-openshift-ca\") pod \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " Nov 02 23:07:33.428892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:33.428895 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1929b2-4756-4bbe-986c-12d83d83d3c2-kube-api-access-slfb8\" (UniqueName: \"kubernetes.io/projected/9c1929b2-4756-4bbe-986c-12d83d83d3c2-kube-api-access-slfb8\") pod \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " Nov 02 23:07:33.429111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:33.428945 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-xks-ca\") pod \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\" (UID: \"9c1929b2-4756-4bbe-986c-12d83d83d3c2\") " Nov 02 23:07:33.429345 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:33.429308 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-openshift-ca" pod "9c1929b2-4756-4bbe-986c-12d83d83d3c2" (UID: "9c1929b2-4756-4bbe-986c-12d83d83d3c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:07:33.429465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:33.429414 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:33.431123 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:33.431093 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-xks-ca" pod "9c1929b2-4756-4bbe-986c-12d83d83d3c2" (UID: "9c1929b2-4756-4bbe-986c-12d83d83d3c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:07:33.431359 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:33.431333 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1929b2-4756-4bbe-986c-12d83d83d3c2-kube-api-access-slfb8" pod "9c1929b2-4756-4bbe-986c-12d83d83d3c2" (UID: "9c1929b2-4756-4bbe-986c-12d83d83d3c2"). InnerVolumeSpecName "kube-api-access-slfb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:07:33.530700 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:33.530668 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfb8\" (UniqueName: \"kubernetes.io/projected/9c1929b2-4756-4bbe-986c-12d83d83d3c2-kube-api-access-slfb8\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:33.530700 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:33.530703 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1929b2-4756-4bbe-986c-12d83d83d3c2-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:33.581683 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:33.581651 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:07:34.208071 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:34.208036 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" Nov 02 23:07:34.208071 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:34.208063 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2" event={"ID":"9c1929b2-4756-4bbe-986c-12d83d83d3c2","Type":"ContainerDied","Data":"c795a8f7d9593c6cb5e82ae822ba97d1a3225b48848ed5ff985c665be7db84ba"} Nov 02 23:07:34.208299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:34.208112 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c795a8f7d9593c6cb5e82ae822ba97d1a3225b48848ed5ff985c665be7db84ba" Nov 02 23:07:36.834974 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.834931 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5"] Nov 02 23:07:36.835539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.835519 5672 cpu_manager.go:414] "containerMap: removing container" podUID="9c1929b2-4756-4bbe-986c-12d83d83d3c2" containerName="revoke-keys" Nov 02 23:07:36.835619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.835541 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1929b2-4756-4bbe-986c-12d83d83d3c2" containerName="revoke-keys" Nov 02 23:07:36.835619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.835571 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" Nov 02 23:07:36.835619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.835579 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd55d7c-49eb-40d2-9daf-49dd2996bc6b" containerName="istio-proxy" Nov 02 23:07:36.835766 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.835701 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1929b2-4756-4bbe-986c-12d83d83d3c2" containerName="revoke-keys" Nov 02 23:07:36.839109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.839086 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:36.889473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.889439 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:07:36.935561 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.935528 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:07:36.935717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.935619 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:07:36.935758 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.935727 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:07:36.937824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.937803 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5"] Nov 02 23:07:36.962135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.962109 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad3b5b2-zdcq5\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:36.962308 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.962172 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vgw\" (UniqueName: \"kubernetes.io/projected/5dfa24b3-ffd4-4893-a002-f05155afb551-kube-api-access-89vgw\") pod \"maas-api-revoke-sub-3ad3b5b2-zdcq5\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:36.962371 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:36.962352 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad3b5b2-zdcq5\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:37.063802 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:37.063767 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad3b5b2-zdcq5\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:37.063971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:37.063813 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad3b5b2-zdcq5\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:37.063971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:37.063845 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vgw\" (UniqueName: \"kubernetes.io/projected/5dfa24b3-ffd4-4893-a002-f05155afb551-kube-api-access-89vgw\") pod \"maas-api-revoke-sub-3ad3b5b2-zdcq5\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:37.064507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:37.064483 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad3b5b2-zdcq5\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:37.066192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:37.066170 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad3b5b2-zdcq5\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:37.116847 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:37.116786 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vgw\" (UniqueName: \"kubernetes.io/projected/5dfa24b3-ffd4-4893-a002-f05155afb551-kube-api-access-89vgw\") pod \"maas-api-revoke-sub-3ad3b5b2-zdcq5\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:37.151094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:37.151067 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:37.517679 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:37.517654 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5"] Nov 02 23:07:37.519329 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:07:37.519302 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfa24b3_ffd4_4893_a002_f05155afb551.slice/crio-bc55c41fc571ae022de4aae3b701988ae7d7728c792ebb78d2be3b31c50f212d WatchSource:0}: Error finding container bc55c41fc571ae022de4aae3b701988ae7d7728c792ebb78d2be3b31c50f212d: Status 404 returned error can't find the container with id bc55c41fc571ae022de4aae3b701988ae7d7728c792ebb78d2be3b31c50f212d Nov 02 23:07:38.236656 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:38.236615 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" event={"ID":"5dfa24b3-ffd4-4893-a002-f05155afb551","Type":"ContainerStarted","Data":"af4c668dc70796fd542b75564d145f31ecf68e4cf5e7b98e0bcc7d2dc4c96a88"} Nov 02 23:07:38.237034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:38.236676 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" event={"ID":"5dfa24b3-ffd4-4893-a002-f05155afb551","Type":"ContainerStarted","Data":"bc55c41fc571ae022de4aae3b701988ae7d7728c792ebb78d2be3b31c50f212d"} Nov 02 23:07:38.346480 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:38.346411 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" podStartSLOduration=2.346393408 podStartE2EDuration="2.346393408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:07:38.343541515 +0000 UTC m=+851.750853632" watchObservedRunningTime="2026-11-02 23:07:38.346393408 +0000 UTC m=+851.753705238" Nov 02 23:07:39.243459 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:39.243415 5672 generic.go:356] "Generic (PLEG): container finished" podID="5dfa24b3-ffd4-4893-a002-f05155afb551" containerID="af4c668dc70796fd542b75564d145f31ecf68e4cf5e7b98e0bcc7d2dc4c96a88" exitCode=0 Nov 02 23:07:39.243912 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:39.243490 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" event={"ID":"5dfa24b3-ffd4-4893-a002-f05155afb551","Type":"ContainerDied","Data":"af4c668dc70796fd542b75564d145f31ecf68e4cf5e7b98e0bcc7d2dc4c96a88"} Nov 02 23:07:40.391184 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:40.391142 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:40.501225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:40.501104 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-openshift-ca\") pod \"5dfa24b3-ffd4-4893-a002-f05155afb551\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " Nov 02 23:07:40.501424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:40.501231 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dfa24b3-ffd4-4893-a002-f05155afb551-kube-api-access-89vgw\" (UniqueName: \"kubernetes.io/projected/5dfa24b3-ffd4-4893-a002-f05155afb551-kube-api-access-89vgw\") pod \"5dfa24b3-ffd4-4893-a002-f05155afb551\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " Nov 02 23:07:40.501424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:40.501302 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-xks-ca\") pod \"5dfa24b3-ffd4-4893-a002-f05155afb551\" (UID: \"5dfa24b3-ffd4-4893-a002-f05155afb551\") " Nov 02 23:07:40.502223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:40.502197 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-openshift-ca" pod "5dfa24b3-ffd4-4893-a002-f05155afb551" (UID: "5dfa24b3-ffd4-4893-a002-f05155afb551"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:07:40.506508 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:40.504104 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfa24b3-ffd4-4893-a002-f05155afb551-kube-api-access-89vgw" pod "5dfa24b3-ffd4-4893-a002-f05155afb551" (UID: "5dfa24b3-ffd4-4893-a002-f05155afb551"). InnerVolumeSpecName "kube-api-access-89vgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:07:40.506508 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:40.504114 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-xks-ca" pod "5dfa24b3-ffd4-4893-a002-f05155afb551" (UID: "5dfa24b3-ffd4-4893-a002-f05155afb551"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:07:40.602794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:40.602765 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89vgw\" (UniqueName: \"kubernetes.io/projected/5dfa24b3-ffd4-4893-a002-f05155afb551-kube-api-access-89vgw\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:40.602794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:40.602791 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:40.602794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:40.602801 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfa24b3-ffd4-4893-a002-f05155afb551-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:41.257455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:41.257410 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" event={"ID":"5dfa24b3-ffd4-4893-a002-f05155afb551","Type":"ContainerDied","Data":"bc55c41fc571ae022de4aae3b701988ae7d7728c792ebb78d2be3b31c50f212d"} Nov 02 23:07:41.257606 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:41.257474 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc55c41fc571ae022de4aae3b701988ae7d7728c792ebb78d2be3b31c50f212d" Nov 02 23:07:41.257606 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:41.257432 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5" Nov 02 23:07:47.559559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.559516 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w"] Nov 02 23:07:47.560174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.560134 5672 cpu_manager.go:414] "containerMap: removing container" podUID="5dfa24b3-ffd4-4893-a002-f05155afb551" containerName="revoke-keys" Nov 02 23:07:47.560174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.560168 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dfa24b3-ffd4-4893-a002-f05155afb551" containerName="revoke-keys" Nov 02 23:07:47.560315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.560303 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dfa24b3-ffd4-4893-a002-f05155afb551" containerName="revoke-keys" Nov 02 23:07:47.564283 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.564260 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:47.614122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.614090 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w"] Nov 02 23:07:47.657906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.657861 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:07:47.658073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.658053 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:07:47.659499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.659479 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:07:47.659583 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.659498 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:07:47.671089 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.671063 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c46241-jzv5w\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:47.671258 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.671239 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhpcg\" (UniqueName: \"kubernetes.io/projected/aacbcf11-86bd-4805-bef1-49fbf14451ff-kube-api-access-zhpcg\") pod \"maas-api-revoke-sub-d1c46241-jzv5w\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:47.671352 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.671330 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c46241-jzv5w\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:47.772454 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.772417 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhpcg\" (UniqueName: \"kubernetes.io/projected/aacbcf11-86bd-4805-bef1-49fbf14451ff-kube-api-access-zhpcg\") pod \"maas-api-revoke-sub-d1c46241-jzv5w\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:47.772627 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.772468 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c46241-jzv5w\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:47.772627 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.772540 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c46241-jzv5w\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:47.773331 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.773302 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1c46241-jzv5w\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:47.774974 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.774945 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1c46241-jzv5w\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:47.825334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.825262 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhpcg\" (UniqueName: \"kubernetes.io/projected/aacbcf11-86bd-4805-bef1-49fbf14451ff-kube-api-access-zhpcg\") pod \"maas-api-revoke-sub-d1c46241-jzv5w\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:47.880106 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:47.880079 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:48.251955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:48.251922 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w"] Nov 02 23:07:48.253772 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:07:48.253741 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacbcf11_86bd_4805_bef1_49fbf14451ff.slice/crio-d416ab8377cba194b1800562045709249aa5790494206593720a0ac11a239a5a WatchSource:0}: Error finding container d416ab8377cba194b1800562045709249aa5790494206593720a0ac11a239a5a: Status 404 returned error can't find the container with id d416ab8377cba194b1800562045709249aa5790494206593720a0ac11a239a5a Nov 02 23:07:48.298571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:48.298545 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" event={"ID":"aacbcf11-86bd-4805-bef1-49fbf14451ff","Type":"ContainerStarted","Data":"d416ab8377cba194b1800562045709249aa5790494206593720a0ac11a239a5a"} Nov 02 23:07:49.306754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:49.306720 5672 generic.go:356] "Generic (PLEG): container finished" podID="aacbcf11-86bd-4805-bef1-49fbf14451ff" containerID="f834794891e434edc85cf16a98ef52b792c2364155fd943e3282f3f3519331d4" exitCode=0 Nov 02 23:07:49.307093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:49.306794 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" event={"ID":"aacbcf11-86bd-4805-bef1-49fbf14451ff","Type":"ContainerDied","Data":"f834794891e434edc85cf16a98ef52b792c2364155fd943e3282f3f3519331d4"} Nov 02 23:07:50.450759 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:50.450734 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:50.600315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:50.600237 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-xks-ca\") pod \"aacbcf11-86bd-4805-bef1-49fbf14451ff\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " Nov 02 23:07:50.600470 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:50.600347 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-openshift-ca\") pod \"aacbcf11-86bd-4805-bef1-49fbf14451ff\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " Nov 02 23:07:50.600470 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:50.600404 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacbcf11-86bd-4805-bef1-49fbf14451ff-kube-api-access-zhpcg\" (UniqueName: \"kubernetes.io/projected/aacbcf11-86bd-4805-bef1-49fbf14451ff-kube-api-access-zhpcg\") pod \"aacbcf11-86bd-4805-bef1-49fbf14451ff\" (UID: \"aacbcf11-86bd-4805-bef1-49fbf14451ff\") " Nov 02 23:07:50.600763 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:50.600730 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-openshift-ca" pod "aacbcf11-86bd-4805-bef1-49fbf14451ff" (UID: "aacbcf11-86bd-4805-bef1-49fbf14451ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:07:50.602477 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:50.602451 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-xks-ca" pod "aacbcf11-86bd-4805-bef1-49fbf14451ff" (UID: "aacbcf11-86bd-4805-bef1-49fbf14451ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:07:50.602477 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:50.602447 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacbcf11-86bd-4805-bef1-49fbf14451ff-kube-api-access-zhpcg" pod "aacbcf11-86bd-4805-bef1-49fbf14451ff" (UID: "aacbcf11-86bd-4805-bef1-49fbf14451ff"). InnerVolumeSpecName "kube-api-access-zhpcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:07:50.701351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:50.701321 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:50.701351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:50.701345 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhpcg\" (UniqueName: \"kubernetes.io/projected/aacbcf11-86bd-4805-bef1-49fbf14451ff-kube-api-access-zhpcg\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:50.701351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:50.701355 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacbcf11-86bd-4805-bef1-49fbf14451ff-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:07:51.318896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:51.318863 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" Nov 02 23:07:51.319094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:51.318899 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w" event={"ID":"aacbcf11-86bd-4805-bef1-49fbf14451ff","Type":"ContainerDied","Data":"d416ab8377cba194b1800562045709249aa5790494206593720a0ac11a239a5a"} Nov 02 23:07:51.319094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:07:51.318955 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d416ab8377cba194b1800562045709249aa5790494206593720a0ac11a239a5a" Nov 02 23:07:56.019995 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:07:56.019962 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:08:04.382128 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:04.382093 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:08:17.855575 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.855539 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv"] Nov 02 23:08:17.855983 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.855955 5672 cpu_manager.go:414] "containerMap: removing container" podUID="aacbcf11-86bd-4805-bef1-49fbf14451ff" containerName="revoke-keys" Nov 02 23:08:17.855983 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.855966 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacbcf11-86bd-4805-bef1-49fbf14451ff" containerName="revoke-keys" Nov 02 23:08:17.856072 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.856049 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacbcf11-86bd-4805-bef1-49fbf14451ff" containerName="revoke-keys" Nov 02 23:08:17.859201 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.859181 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:17.912519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.912482 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv"] Nov 02 23:08:17.955484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.955458 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:08:17.955642 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.955455 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:08:17.955642 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.955501 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:08:17.955642 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.955583 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:08:17.955899 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.955871 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbfee3a3-657fv\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:17.956039 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.955918 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hzp\" (UniqueName: \"kubernetes.io/projected/0988e692-a3ab-4478-a116-7a50b50113f1-kube-api-access-82hzp\") pod \"maas-api-revoke-sub-cbfee3a3-657fv\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:17.956101 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:17.956062 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbfee3a3-657fv\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:18.057189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:18.057135 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbfee3a3-657fv\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:18.057350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:18.057251 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbfee3a3-657fv\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:18.057350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:18.057298 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hzp\" (UniqueName: \"kubernetes.io/projected/0988e692-a3ab-4478-a116-7a50b50113f1-kube-api-access-82hzp\") pod \"maas-api-revoke-sub-cbfee3a3-657fv\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:18.058000 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:18.057975 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbfee3a3-657fv\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:18.059554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:18.059532 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbfee3a3-657fv\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:18.110620 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:18.110554 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hzp\" (UniqueName: \"kubernetes.io/projected/0988e692-a3ab-4478-a116-7a50b50113f1-kube-api-access-82hzp\") pod \"maas-api-revoke-sub-cbfee3a3-657fv\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:18.171729 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:18.171702 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:18.535629 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:18.535601 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv"] Nov 02 23:08:18.537366 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:08:18.537335 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0988e692_a3ab_4478_a116_7a50b50113f1.slice/crio-ac01db53d7ca90a3d926c73cb19d0f76cec8121e49be66a2205f16928001fcf1 WatchSource:0}: Error finding container ac01db53d7ca90a3d926c73cb19d0f76cec8121e49be66a2205f16928001fcf1: Status 404 returned error can't find the container with id ac01db53d7ca90a3d926c73cb19d0f76cec8121e49be66a2205f16928001fcf1 Nov 02 23:08:19.487060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:19.487027 5672 generic.go:356] "Generic (PLEG): container finished" podID="0988e692-a3ab-4478-a116-7a50b50113f1" containerID="2afcfd3195d5a8998996d3d054155224abd859802c8f535cf726f9f7c70b1d97" exitCode=0 Nov 02 23:08:19.487495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:19.487125 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" event={"ID":"0988e692-a3ab-4478-a116-7a50b50113f1","Type":"ContainerDied","Data":"2afcfd3195d5a8998996d3d054155224abd859802c8f535cf726f9f7c70b1d97"} Nov 02 23:08:19.487495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:19.487203 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" event={"ID":"0988e692-a3ab-4478-a116-7a50b50113f1","Type":"ContainerStarted","Data":"ac01db53d7ca90a3d926c73cb19d0f76cec8121e49be66a2205f16928001fcf1"} Nov 02 23:08:20.632836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:20.632812 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:20.783324 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:20.783245 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-openshift-ca\") pod \"0988e692-a3ab-4478-a116-7a50b50113f1\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " Nov 02 23:08:20.783324 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:20.783305 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-xks-ca\") pod \"0988e692-a3ab-4478-a116-7a50b50113f1\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " Nov 02 23:08:20.783536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:20.783403 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0988e692-a3ab-4478-a116-7a50b50113f1-kube-api-access-82hzp\" (UniqueName: \"kubernetes.io/projected/0988e692-a3ab-4478-a116-7a50b50113f1-kube-api-access-82hzp\") pod \"0988e692-a3ab-4478-a116-7a50b50113f1\" (UID: \"0988e692-a3ab-4478-a116-7a50b50113f1\") " Nov 02 23:08:20.783675 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:20.783649 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-openshift-ca" pod "0988e692-a3ab-4478-a116-7a50b50113f1" (UID: "0988e692-a3ab-4478-a116-7a50b50113f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:08:20.785575 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:20.785551 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-xks-ca" pod "0988e692-a3ab-4478-a116-7a50b50113f1" (UID: "0988e692-a3ab-4478-a116-7a50b50113f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:08:20.785772 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:20.785747 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0988e692-a3ab-4478-a116-7a50b50113f1-kube-api-access-82hzp" pod "0988e692-a3ab-4478-a116-7a50b50113f1" (UID: "0988e692-a3ab-4478-a116-7a50b50113f1"). InnerVolumeSpecName "kube-api-access-82hzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:08:20.884278 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:20.884238 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82hzp\" (UniqueName: \"kubernetes.io/projected/0988e692-a3ab-4478-a116-7a50b50113f1-kube-api-access-82hzp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:08:20.884278 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:20.884275 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:08:20.884520 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:20.884291 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0988e692-a3ab-4478-a116-7a50b50113f1-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:08:21.183409 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.183300 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz"] Nov 02 23:08:21.183834 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.183815 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0988e692-a3ab-4478-a116-7a50b50113f1" containerName="revoke-keys" Nov 02 23:08:21.183834 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.183833 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0988e692-a3ab-4478-a116-7a50b50113f1" containerName="revoke-keys" Nov 02 23:08:21.183961 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.183894 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0988e692-a3ab-4478-a116-7a50b50113f1" containerName="revoke-keys" Nov 02 23:08:21.187103 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.187084 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.240193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.240142 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz"] Nov 02 23:08:21.283276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.283219 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"kube-root-ca.crt\"" Nov 02 23:08:21.283430 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.283398 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"e2e-conflict-a-11bf0f82-kserve-self-signed-certs\"" Nov 02 23:08:21.283511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.283445 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"default-dockercfg-66dv6\"" Nov 02 23:08:21.283511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.283446 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"openshift-service-ca.crt\"" Nov 02 23:08:21.288572 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.288551 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tmp-dir\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.288676 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.288587 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-model-cache\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.288676 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.288609 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-home\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.288773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.288761 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tls-certs\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.288826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.288814 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zjqz\" (UniqueName: \"kubernetes.io/projected/fdba3b13-1172-4ec7-bb34-b882a753f6f6-kube-api-access-6zjqz\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.288865 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.288839 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-dshm\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.390168 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.390117 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-home\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.390374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.390204 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tls-certs\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.390374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.390254 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zjqz\" (UniqueName: \"kubernetes.io/projected/fdba3b13-1172-4ec7-bb34-b882a753f6f6-kube-api-access-6zjqz\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.390374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.390287 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-dshm\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.390374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.390321 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tmp-dir\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.390374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.390343 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-model-cache\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.390677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.390639 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-home\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.390737 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.390687 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tmp-dir\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.390829 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.390801 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-model-cache\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.392632 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.392604 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-dshm\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.393105 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.393086 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tls-certs\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.447902 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.447828 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zjqz\" (UniqueName: \"kubernetes.io/projected/fdba3b13-1172-4ec7-bb34-b882a753f6f6-kube-api-access-6zjqz\") pod \"e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.499443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.499410 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:21.500990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.500961 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" event={"ID":"0988e692-a3ab-4478-a116-7a50b50113f1","Type":"ContainerDied","Data":"ac01db53d7ca90a3d926c73cb19d0f76cec8121e49be66a2205f16928001fcf1"} Nov 02 23:08:21.501133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.500992 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv" Nov 02 23:08:21.501133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.501015 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac01db53d7ca90a3d926c73cb19d0f76cec8121e49be66a2205f16928001fcf1" Nov 02 23:08:21.873510 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:21.873484 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz"] Nov 02 23:08:21.874784 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:08:21.874742 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdba3b13_1172_4ec7_bb34_b882a753f6f6.slice/crio-49728a20be63201dde53bfe5db1b5fcb18e7cfd479dfb4010cb6217110e8a220 WatchSource:0}: Error finding container 49728a20be63201dde53bfe5db1b5fcb18e7cfd479dfb4010cb6217110e8a220: Status 404 returned error can't find the container with id 49728a20be63201dde53bfe5db1b5fcb18e7cfd479dfb4010cb6217110e8a220 Nov 02 23:08:22.511164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:22.511111 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" event={"ID":"fdba3b13-1172-4ec7-bb34-b882a753f6f6","Type":"ContainerStarted","Data":"0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5"} Nov 02 23:08:22.511370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:22.511172 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" event={"ID":"fdba3b13-1172-4ec7-bb34-b882a753f6f6","Type":"ContainerStarted","Data":"49728a20be63201dde53bfe5db1b5fcb18e7cfd479dfb4010cb6217110e8a220"} Nov 02 23:08:22.619785 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:22.619728 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" podStartSLOduration=1.619714295 podStartE2EDuration="1.619714295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:22.617319 +0000 UTC m=+896.024630831" watchObservedRunningTime="2026-11-02 23:08:22.619714295 +0000 UTC m=+896.027026125" Nov 02 23:08:23.499641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:23.499607 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:26.020269 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:08:26.020228 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:08:30.468722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.468689 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh"] Nov 02 23:08:30.472855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.472830 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.515698 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.515671 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf"] Nov 02 23:08:30.519114 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.519098 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.568908 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.568877 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh"] Nov 02 23:08:30.580523 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.580496 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.580693 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.580539 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.580693 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.580569 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.580816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.580775 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.580893 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.580874 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.580943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.580902 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr585\" (UniqueName: \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-kube-api-access-tr585\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.580943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.580920 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.580943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.580934 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.581056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.580979 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.616947 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.616914 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf"] Nov 02 23:08:30.682450 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.682417 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.682618 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.682455 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.682618 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.682477 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.682618 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.682559 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.682618 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.682615 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.682832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.682651 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tzbk\" (UniqueName: \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-kube-api-access-4tzbk\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.682896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.682827 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.682896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.682863 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr585\" (UniqueName: \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-kube-api-access-tr585\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.682896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.682892 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.683050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.682919 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.683050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.682990 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.683050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.683040 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.683236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.683065 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.683236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.683120 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.683236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.683168 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.683406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.683263 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.683406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.683320 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.683502 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.683485 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.683502 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.683493 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.683911 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.683456 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.683911 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.683877 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.684092 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.683917 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.684298 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.684279 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.686094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.686061 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.686245 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.686139 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.736487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.736418 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr585\" (UniqueName: \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-kube-api-access-tr585\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.736620 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.736501 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vf4wh\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.784658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.784631 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tzbk\" (UniqueName: \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-kube-api-access-4tzbk\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.784805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.784677 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.784805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.784703 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.784805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.784724 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.784805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.784740 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.784805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.784765 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.784805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.784789 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.785115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.784834 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.785115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.784921 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.785250 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.785222 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.785303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.785269 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.785362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.785340 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.785416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.785381 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.785741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.785723 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.786629 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.786613 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:30.787125 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.787099 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.787534 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.787512 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.839546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.839496 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tzbk\" (UniqueName: \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-kube-api-access-4tzbk\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:30.840028 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:30.840008 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-4rwcf\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:31.131814 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.131739 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:31.203416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.203380 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh"] Nov 02 23:08:31.207767 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:08:31.207730 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6eccd0b_f56a_4750_bdc8_6241fc80ea4d.slice/crio-9c26278e66594d439f36d23bca2dd83d250bd0b9a1dcb442ef1241ab29b66a17 WatchSource:0}: Error finding container 9c26278e66594d439f36d23bca2dd83d250bd0b9a1dcb442ef1241ab29b66a17: Status 404 returned error can't find the container with id 9c26278e66594d439f36d23bca2dd83d250bd0b9a1dcb442ef1241ab29b66a17 Nov 02 23:08:31.210574 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.210534 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:31.210663 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.210630 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:31.210712 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.210695 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:31.499684 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.499655 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:31.505627 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.505601 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf"] Nov 02 23:08:31.508267 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:08:31.508234 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b474fc_c267_4b2d_b2f0_72340c7ecc69.slice/crio-d35ddd478655d130cbcd00cf441d448501540382175bc2e41c661f1bd48a0a9e WatchSource:0}: Error finding container d35ddd478655d130cbcd00cf441d448501540382175bc2e41c661f1bd48a0a9e: Status 404 returned error can't find the container with id d35ddd478655d130cbcd00cf441d448501540382175bc2e41c661f1bd48a0a9e Nov 02 23:08:31.510416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.510383 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:31.510528 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.510463 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:31.510528 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.510506 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:08:31.523791 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.523771 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:31.585569 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.585508 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" event={"ID":"15b474fc-c267-4b2d-b2f0-72340c7ecc69","Type":"ContainerStarted","Data":"d35ddd478655d130cbcd00cf441d448501540382175bc2e41c661f1bd48a0a9e"} Nov 02 23:08:31.587686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.587651 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" event={"ID":"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d","Type":"ContainerStarted","Data":"37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438"} Nov 02 23:08:31.587800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.587726 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" event={"ID":"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d","Type":"ContainerStarted","Data":"9c26278e66594d439f36d23bca2dd83d250bd0b9a1dcb442ef1241ab29b66a17"} Nov 02 23:08:31.603581 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.603558 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:08:31.787662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.787569 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:31.866623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:31.866563 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" podStartSLOduration=1.8665464040000002 podStartE2EDuration="1.866546404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:31.757603927 +0000 UTC m=+905.164915758" watchObservedRunningTime="2026-11-02 23:08:31.866546404 +0000 UTC m=+905.273858228" Nov 02 23:08:32.595593 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:32.595551 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" event={"ID":"15b474fc-c267-4b2d-b2f0-72340c7ecc69","Type":"ContainerStarted","Data":"0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4"} Nov 02 23:08:32.788870 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:32.788834 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:08:32.789043 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:32.788894 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:08:32.799297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:32.799243 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podStartSLOduration=2.799230514 podStartE2EDuration="2.799230514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:32.751287268 +0000 UTC m=+906.158599126" watchObservedRunningTime="2026-11-02 23:08:32.799230514 +0000 UTC m=+906.206542344" Nov 02 23:08:33.136762 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:33.134478 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:33.143202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:33.137257 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:08:33.143202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:33.137318 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 23:08:33.788645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:33.788495 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:08:33.788645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:33.788567 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:08:34.150229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:34.136492 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:08:34.150229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:34.136554 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 23:08:34.787674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:34.787582 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:08:34.787674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:34.787661 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:08:35.132594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:35.132502 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:08:35.133199 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:35.133135 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 23:08:35.788229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:35.787487 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:08:35.788229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:35.787544 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:08:36.143229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:36.136584 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:08:36.143229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:36.136656 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 23:08:36.788124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:36.787587 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:08:36.788124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:36.787650 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:08:37.136373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:37.136266 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:08:37.136677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:37.136625 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 23:08:37.218604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:37.218568 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:08:37.786911 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:37.786874 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:08:37.787097 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:37.786939 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:08:38.132747 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:38.132656 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:08:38.132747 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:38.132734 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 23:08:38.787921 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:38.787883 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 23:08:38.788362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:38.787951 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 23:08:39.132556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:39.132455 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 23:08:39.132556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:39.132532 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 23:08:40.544651 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:40.544619 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:40.556174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:40.548509 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:40.566925 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:40.566699 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:08:41.133422 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:41.133382 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:08:41.133632 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:41.133449 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:08:42.175352 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:42.175294 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:08:42.175983 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:42.175928 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:08:43.217339 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:43.217294 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:08:43.217778 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:43.217372 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:08:43.433494 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:43.433462 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:43.433884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:43.433855 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:43.451033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:43.451009 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:08:52.328687 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.328653 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92"] Nov 02 23:08:52.332465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.332443 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.428915 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.428880 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ba1918\"/\"distinct-w2-ba1918-kserve-self-signed-certs\"" Nov 02 23:08:52.431432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.431404 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92"] Nov 02 23:08:52.475368 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.475337 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-tmp-dir\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.475533 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.475404 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e279ff-eba6-4783-ac4c-dbc824466aa2-tls-certs\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.475533 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.475508 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-model-cache\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.475638 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.475606 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-home\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.475691 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.475642 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-dshm\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.475691 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.475668 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbf7v\" (UniqueName: \"kubernetes.io/projected/60e279ff-eba6-4783-ac4c-dbc824466aa2-kube-api-access-fbf7v\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.576453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.576395 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e279ff-eba6-4783-ac4c-dbc824466aa2-tls-certs\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.576453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.576454 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-model-cache\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.576709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.576530 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-home\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.576709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.576576 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-dshm\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.576709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.576612 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbf7v\" (UniqueName: \"kubernetes.io/projected/60e279ff-eba6-4783-ac4c-dbc824466aa2-kube-api-access-fbf7v\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.576709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.576637 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-tmp-dir\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.576923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.576859 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-model-cache\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.576923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.576887 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-home\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.577103 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.577083 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-tmp-dir\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.578928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.578854 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-dshm\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.579217 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.579200 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e279ff-eba6-4783-ac4c-dbc824466aa2-tls-certs\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.630339 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.630301 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbf7v\" (UniqueName: \"kubernetes.io/projected/60e279ff-eba6-4783-ac4c-dbc824466aa2-kube-api-access-fbf7v\") pod \"distinct-w2-ba1918-kserve-75bcd4968f-phg92\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:52.645169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:52.645117 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:53.027650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:53.027621 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92"] Nov 02 23:08:53.028585 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:08:53.028555 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60e279ff_eba6_4783_ac4c_dbc824466aa2.slice/crio-1883490c10322f707aa71f16886688424ea66efa66f6375e4d1e2b439aa64584 WatchSource:0}: Error finding container 1883490c10322f707aa71f16886688424ea66efa66f6375e4d1e2b439aa64584: Status 404 returned error can't find the container with id 1883490c10322f707aa71f16886688424ea66efa66f6375e4d1e2b439aa64584 Nov 02 23:08:53.753227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:53.753186 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" event={"ID":"60e279ff-eba6-4783-ac4c-dbc824466aa2","Type":"ContainerStarted","Data":"79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b"} Nov 02 23:08:53.753726 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:53.753238 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" event={"ID":"60e279ff-eba6-4783-ac4c-dbc824466aa2","Type":"ContainerStarted","Data":"1883490c10322f707aa71f16886688424ea66efa66f6375e4d1e2b439aa64584"} Nov 02 23:08:53.862029 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:53.861976 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podStartSLOduration=1.861962745 podStartE2EDuration="1.861962745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:53.859638261 +0000 UTC m=+927.266950094" watchObservedRunningTime="2026-11-02 23:08:53.861962745 +0000 UTC m=+927.269274574" Nov 02 23:08:54.646025 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:54.645990 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:08:56.020086 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:08:56.020043 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:08:59.943919 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:59.943885 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts"] Nov 02 23:08:59.947812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:59.947792 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:08:59.996788 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:08:59.996764 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fb6e1d-openshift-default-dockercfg-5lz5h\"" Nov 02 23:09:00.009710 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.009681 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:09:00.045936 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.045898 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts"] Nov 02 23:09:00.050383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.050337 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.050383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.050382 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.050609 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.050484 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.050609 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.050525 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.050732 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.050632 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.050732 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.050662 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdf9\" (UniqueName: \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-kube-api-access-6zdf9\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.050834 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.050796 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a976591-e56a-4344-afbf-1a0c369a10d1-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.050890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.050837 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.050890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.050860 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.151969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.151937 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.151969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.151971 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.152254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.152038 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.152254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.152059 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.152254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.152084 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.152254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.152114 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.152254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.152203 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.152254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.152237 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdf9\" (UniqueName: \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-kube-api-access-6zdf9\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.152646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.152308 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a976591-e56a-4344-afbf-1a0c369a10d1-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.152646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.152467 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.153036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.153004 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a976591-e56a-4344-afbf-1a0c369a10d1-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.153338 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.153307 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.153545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.153521 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.153690 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.153559 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.154611 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.154587 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.154822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.154803 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.205610 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.205549 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.205722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.205704 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdf9\" (UniqueName: \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-kube-api-access-6zdf9\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.262836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.262805 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:00.681784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.681748 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts"] Nov 02 23:09:00.682808 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:09:00.682775 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a976591_e56a_4344_afbf_1a0c369a10d1.slice/crio-dc49dfebead99ab8399da17def0018caf6ba63cfe3a25a5ff7eb8c2ecaddcbdd WatchSource:0}: Error finding container dc49dfebead99ab8399da17def0018caf6ba63cfe3a25a5ff7eb8c2ecaddcbdd: Status 404 returned error can't find the container with id dc49dfebead99ab8399da17def0018caf6ba63cfe3a25a5ff7eb8c2ecaddcbdd Nov 02 23:09:00.685258 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.685223 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:09:00.685334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.685289 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:09:00.685334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.685322 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:09:00.803109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.803070 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" event={"ID":"5a976591-e56a-4344-afbf-1a0c369a10d1","Type":"ContainerStarted","Data":"b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de"} Nov 02 23:09:00.803432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.803413 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" event={"ID":"5a976591-e56a-4344-afbf-1a0c369a10d1","Type":"ContainerStarted","Data":"dc49dfebead99ab8399da17def0018caf6ba63cfe3a25a5ff7eb8c2ecaddcbdd"} Nov 02 23:09:00.916102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:00.916046 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" podStartSLOduration=1.916029388 podStartE2EDuration="1.916029388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:00.91293007 +0000 UTC m=+934.320241911" watchObservedRunningTime="2026-11-02 23:09:00.916029388 +0000 UTC m=+934.323341215" Nov 02 23:09:01.263679 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:01.263627 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:01.265389 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:01.265358 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:09:01.265638 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:01.265603 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:09:02.264173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:02.263547 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:09:02.264173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:02.263609 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:09:02.650015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:02.649921 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:09:03.275173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:03.274589 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:09:03.275173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:03.274656 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:09:03.351975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:03.351927 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:09:04.112340 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:04.112282 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:09:04.267225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:04.265073 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:09:04.267225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:04.265167 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:09:04.356285 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:04.354233 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:09:04.445227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:04.434647 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:09:05.264036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:05.263995 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:09:05.264286 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:05.264052 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:09:06.264054 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:06.264020 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:09:06.264602 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:06.264072 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:09:07.263919 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:07.263883 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 23:09:07.264220 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:07.263935 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 23:09:08.268639 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:08.268605 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:08.269028 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:08.268868 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:08.269533 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:08.269511 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:09:25.996034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:25.995993 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt"] Nov 02 23:09:25.999768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:25.999748 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.021378 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:09:26.021349 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:09:26.094414 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.094378 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ba1918-openshift-default-dockercfg-k6bc8\"" Nov 02 23:09:26.097019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.096989 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt"] Nov 02 23:09:26.132442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.132411 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s277\" (UniqueName: \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-kube-api-access-9s277\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.132574 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.132452 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.132574 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.132476 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a7e949-3e53-406b-83ec-9680954ad105-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.132663 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.132588 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.132663 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.132618 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.132731 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.132683 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.132773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.132737 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.132773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.132757 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a7e949-3e53-406b-83ec-9680954ad105-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.132876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.132863 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.233497 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.233464 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.233497 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.233502 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s277\" (UniqueName: \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-kube-api-access-9s277\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.233705 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.233618 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.233705 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.233661 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a7e949-3e53-406b-83ec-9680954ad105-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.233705 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.233694 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.233816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.233713 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.233816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.233749 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.233901 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.233824 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.233957 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.233924 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a7e949-3e53-406b-83ec-9680954ad105-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.234086 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.234058 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.234239 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.234212 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.234330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.234310 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.234410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.234389 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.234605 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.234586 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a7e949-3e53-406b-83ec-9680954ad105-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.236946 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.236925 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.237088 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.237072 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a7e949-3e53-406b-83ec-9680954ad105-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.286139 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.286074 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.286265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.286168 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s277\" (UniqueName: \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-kube-api-access-9s277\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.314007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.313980 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:26.729257 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.729234 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt"] Nov 02 23:09:26.730268 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:09:26.730236 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a7e949_3e53_406b_83ec_9680954ad105.slice/crio-740b749d3bbd188805237cdd2984cb1bff434179adb86269db090d69f4f37cd5 WatchSource:0}: Error finding container 740b749d3bbd188805237cdd2984cb1bff434179adb86269db090d69f4f37cd5: Status 404 returned error can't find the container with id 740b749d3bbd188805237cdd2984cb1bff434179adb86269db090d69f4f37cd5 Nov 02 23:09:26.732462 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.732430 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:09:26.732545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.732492 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:09:26.732545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.732522 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:09:26.999761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.999675 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" event={"ID":"09a7e949-3e53-406b-83ec-9680954ad105","Type":"ContainerStarted","Data":"77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0"} Nov 02 23:09:26.999761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:26.999753 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" event={"ID":"09a7e949-3e53-406b-83ec-9680954ad105","Type":"ContainerStarted","Data":"740b749d3bbd188805237cdd2984cb1bff434179adb86269db090d69f4f37cd5"} Nov 02 23:09:27.114247 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:27.114138 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" podStartSLOduration=2.114121423 podStartE2EDuration="2.114121423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:27.112616837 +0000 UTC m=+960.519928670" watchObservedRunningTime="2026-11-02 23:09:27.114121423 +0000 UTC m=+960.521433253" Nov 02 23:09:27.315166 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:27.315066 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:27.316674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:27.316648 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 23:09:27.316796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:27.316714 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 23:09:28.314427 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:28.314390 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 23:09:28.314916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:28.314454 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 23:09:29.314849 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:29.314812 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 23:09:29.315281 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:29.314878 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 23:09:30.315047 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:30.315014 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 23:09:30.315451 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:30.315073 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 23:09:31.314639 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:31.314602 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 23:09:31.314842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:31.314674 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 23:09:32.314590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:32.314553 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 23:09:32.314971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:32.314630 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 23:09:33.315346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:33.315307 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 23:09:33.315725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:33.315409 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 23:09:34.318564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:34.318532 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:34.318947 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:34.318927 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:34.319503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:34.319483 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:09:35.197466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:09:35.197434 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:09:56.020203 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:09:56.020162 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:10:01.809695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:01.809650 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:10:02.420483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.420448 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb"] Nov 02 23:10:02.424615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.424584 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.496628 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.496597 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb"] Nov 02 23:10:02.520895 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.520868 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ba1918\"/\"distinct-2-w2-ba1918-kserve-self-signed-certs\"" Nov 02 23:10:02.523365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.523342 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4278c7e1-9584-47f4-ac3d-323867b57b83-tls-certs\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.523555 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.523374 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkttz\" (UniqueName: \"kubernetes.io/projected/4278c7e1-9584-47f4-ac3d-323867b57b83-kube-api-access-hkttz\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.523555 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.523466 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-model-cache\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.523555 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.523517 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-dshm\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.523688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.523570 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-home\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.523688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.523653 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-tmp-dir\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.624473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.624426 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-tmp-dir\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.624677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.624503 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4278c7e1-9584-47f4-ac3d-323867b57b83-tls-certs\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.624677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.624536 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkttz\" (UniqueName: \"kubernetes.io/projected/4278c7e1-9584-47f4-ac3d-323867b57b83-kube-api-access-hkttz\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.624677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.624595 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-model-cache\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.624677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.624642 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-dshm\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.624926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.624717 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-home\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.625175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.625127 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-home\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.625408 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.625389 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-tmp-dir\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.625886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.625846 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-model-cache\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.628533 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.628510 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4278c7e1-9584-47f4-ac3d-323867b57b83-tls-certs\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.638511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.638443 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-dshm\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.680129 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.680055 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkttz\" (UniqueName: \"kubernetes.io/projected/4278c7e1-9584-47f4-ac3d-323867b57b83-kube-api-access-hkttz\") pod \"distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:02.738362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:02.738322 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:03.149373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:03.149348 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb"] Nov 02 23:10:03.151478 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:10:03.151444 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4278c7e1_9584_47f4_ac3d_323867b57b83.slice/crio-f12e99dd9debd1125606df86e4d4296072ba0221fa57fb6460e222510d15fcce WatchSource:0}: Error finding container f12e99dd9debd1125606df86e4d4296072ba0221fa57fb6460e222510d15fcce: Status 404 returned error can't find the container with id f12e99dd9debd1125606df86e4d4296072ba0221fa57fb6460e222510d15fcce Nov 02 23:10:03.243641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:03.243601 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" event={"ID":"4278c7e1-9584-47f4-ac3d-323867b57b83","Type":"ContainerStarted","Data":"b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708"} Nov 02 23:10:03.243791 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:03.243657 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" event={"ID":"4278c7e1-9584-47f4-ac3d-323867b57b83","Type":"ContainerStarted","Data":"f12e99dd9debd1125606df86e4d4296072ba0221fa57fb6460e222510d15fcce"} Nov 02 23:10:03.352032 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:03.351964 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podStartSLOduration=1.351944203 podStartE2EDuration="1.351944203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:03.350188142 +0000 UTC m=+996.757499972" watchObservedRunningTime="2026-11-02 23:10:03.351944203 +0000 UTC m=+996.759256034" Nov 02 23:10:03.741314 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:03.741280 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:12.738859 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:12.738777 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:12.751090 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:12.751062 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:13.346270 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:13.346240 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:10:26.021487 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:10:26.021447 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:10:26.848732 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:26.848695 5672 scope.go:122] "RemoveContainer" containerID="83f412b30c0123b65cbc80990957b3ece996255ea6788cdca01929461efac7af" Nov 02 23:10:26.858907 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:26.858886 5672 scope.go:122] "RemoveContainer" containerID="dbdeb2bc754fcc315b004c6b0f384d8c06ac10ed10d08b876f7bb7e072ebd613" Nov 02 23:10:26.868262 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:26.868212 5672 scope.go:122] "RemoveContainer" containerID="3bf54c5cb17c676cece9f8ac3cd3493ffe6422f87810b78c3ce3c7fc54db027d" Nov 02 23:10:26.877912 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:26.877894 5672 scope.go:122] "RemoveContainer" containerID="e912498d02bd9ad372cc4f877854c651e9f233ffe99634ffda1e2d25a656279b" Nov 02 23:10:31.738961 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:31.738916 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:10:54.497940 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.497906 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd"] Nov 02 23:10:54.501559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.501536 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.556835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.556803 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-home\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.557011 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.556950 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-tmp-dir\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.557077 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.557066 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-model-cache\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.557136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.557095 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-dshm\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.557212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.557136 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d891d08-d471-4358-9dea-d25276081c2f-tls-certs\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.557212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.557179 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q685s\" (UniqueName: \"kubernetes.io/projected/3d891d08-d471-4358-9dea-d25276081c2f-kube-api-access-q685s\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.598472 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.598439 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3d8404\"/\"distinct-2-w1-3d8404-kserve-self-signed-certs\"" Nov 02 23:10:54.601313 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.601282 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd"] Nov 02 23:10:54.658805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.658765 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-tmp-dir\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.659012 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.658876 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-model-cache\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.659012 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.658912 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-dshm\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.659012 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.658952 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d891d08-d471-4358-9dea-d25276081c2f-tls-certs\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.659012 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.658979 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q685s\" (UniqueName: \"kubernetes.io/projected/3d891d08-d471-4358-9dea-d25276081c2f-kube-api-access-q685s\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.659253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.659072 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-home\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.659356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.659332 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-tmp-dir\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.659836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.659809 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-home\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.660200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.660171 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-model-cache\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.662072 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.662044 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-dshm\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.662596 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.662575 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d891d08-d471-4358-9dea-d25276081c2f-tls-certs\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.712643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.712612 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q685s\" (UniqueName: \"kubernetes.io/projected/3d891d08-d471-4358-9dea-d25276081c2f-kube-api-access-q685s\") pod \"distinct-2-w1-3d8404-kserve-95dd58556-4tcpd\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:54.815491 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:54.815391 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:55.224824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:55.224799 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd"] Nov 02 23:10:55.227596 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:10:55.227561 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d891d08_d471_4358_9dea_d25276081c2f.slice/crio-bd7204818f974a0631fe8394ecfb232ab4dce0b44664d44f59b017822a1562c1 WatchSource:0}: Error finding container bd7204818f974a0631fe8394ecfb232ab4dce0b44664d44f59b017822a1562c1: Status 404 returned error can't find the container with id bd7204818f974a0631fe8394ecfb232ab4dce0b44664d44f59b017822a1562c1 Nov 02 23:10:55.632313 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:55.632212 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" event={"ID":"3d891d08-d471-4358-9dea-d25276081c2f","Type":"ContainerStarted","Data":"3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0"} Nov 02 23:10:55.632313 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:55.632260 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" event={"ID":"3d891d08-d471-4358-9dea-d25276081c2f","Type":"ContainerStarted","Data":"bd7204818f974a0631fe8394ecfb232ab4dce0b44664d44f59b017822a1562c1"} Nov 02 23:10:55.738817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:55.738746 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" podStartSLOduration=1.7387274609999999 podStartE2EDuration="1.738727461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:55.737928225 +0000 UTC m=+1049.145240052" watchObservedRunningTime="2026-11-02 23:10:55.738727461 +0000 UTC m=+1049.146039303" Nov 02 23:10:55.815754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:10:55.815719 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:10:56.021300 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:10:56.021264 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:11:00.762993 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.762946 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g"] Nov 02 23:11:00.767219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.767194 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:00.816325 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.816293 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:11:00.820013 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.819987 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g"] Nov 02 23:11:00.829943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.829914 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a33334a-f6m6g\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:00.830099 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.830084 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4dt6\" (UniqueName: \"kubernetes.io/projected/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-kube-api-access-c4dt6\") pod \"maas-api-revoke-sub-7a33334a-f6m6g\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:00.830209 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.830191 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a33334a-f6m6g\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:00.862513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.862479 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:11:00.862684 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.862569 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:11:00.862684 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.862630 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:11:00.931426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.931399 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4dt6\" (UniqueName: \"kubernetes.io/projected/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-kube-api-access-c4dt6\") pod \"maas-api-revoke-sub-7a33334a-f6m6g\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:00.931608 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.931458 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a33334a-f6m6g\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:00.931608 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.931515 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a33334a-f6m6g\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:00.932199 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.932174 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a33334a-f6m6g\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:00.934007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.933983 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a33334a-f6m6g\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:00.993259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:00.993225 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4dt6\" (UniqueName: \"kubernetes.io/projected/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-kube-api-access-c4dt6\") pod \"maas-api-revoke-sub-7a33334a-f6m6g\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:01.082007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:01.081918 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:01.458529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:01.458502 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g"] Nov 02 23:11:01.462511 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:11:01.462477 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94e21cd_1b1a_4972_8d47_18fb8e3c0856.slice/crio-f78cc76224f15261c9f4a542155459afca9cc428689c5e24db1cde92b790b79d WatchSource:0}: Error finding container f78cc76224f15261c9f4a542155459afca9cc428689c5e24db1cde92b790b79d: Status 404 returned error can't find the container with id f78cc76224f15261c9f4a542155459afca9cc428689c5e24db1cde92b790b79d Nov 02 23:11:01.675650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:01.675616 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" event={"ID":"a94e21cd-1b1a-4972-8d47-18fb8e3c0856","Type":"ContainerStarted","Data":"f78cc76224f15261c9f4a542155459afca9cc428689c5e24db1cde92b790b79d"} Nov 02 23:11:02.683358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:02.683260 5672 generic.go:356] "Generic (PLEG): container finished" podID="a94e21cd-1b1a-4972-8d47-18fb8e3c0856" containerID="65241516b3b1cde08cd8997678758b37a7a5ce393d030eefa8061f79250bf4c4" exitCode=0 Nov 02 23:11:02.683733 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:02.683343 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" event={"ID":"a94e21cd-1b1a-4972-8d47-18fb8e3c0856","Type":"ContainerDied","Data":"65241516b3b1cde08cd8997678758b37a7a5ce393d030eefa8061f79250bf4c4"} Nov 02 23:11:03.845750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:03.845280 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:03.967210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:03.967077 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-xks-ca\") pod \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " Nov 02 23:11:03.967398 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:03.967216 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-openshift-ca\") pod \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " Nov 02 23:11:03.967398 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:03.967375 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-kube-api-access-c4dt6\" (UniqueName: \"kubernetes.io/projected/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-kube-api-access-c4dt6\") pod \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\" (UID: \"a94e21cd-1b1a-4972-8d47-18fb8e3c0856\") " Nov 02 23:11:03.968588 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:03.968558 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-openshift-ca" pod "a94e21cd-1b1a-4972-8d47-18fb8e3c0856" (UID: "a94e21cd-1b1a-4972-8d47-18fb8e3c0856"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:11:03.970531 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:03.970500 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-kube-api-access-c4dt6" pod "a94e21cd-1b1a-4972-8d47-18fb8e3c0856" (UID: "a94e21cd-1b1a-4972-8d47-18fb8e3c0856"). InnerVolumeSpecName "kube-api-access-c4dt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:11:03.970781 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:03.970743 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-xks-ca" pod "a94e21cd-1b1a-4972-8d47-18fb8e3c0856" (UID: "a94e21cd-1b1a-4972-8d47-18fb8e3c0856"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:11:04.069275 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:04.069237 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:04.069275 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:04.069276 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:04.069530 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:04.069292 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4dt6\" (UniqueName: \"kubernetes.io/projected/a94e21cd-1b1a-4972-8d47-18fb8e3c0856-kube-api-access-c4dt6\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:04.697028 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:04.697000 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" Nov 02 23:11:04.697286 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:04.697037 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g" event={"ID":"a94e21cd-1b1a-4972-8d47-18fb8e3c0856","Type":"ContainerDied","Data":"f78cc76224f15261c9f4a542155459afca9cc428689c5e24db1cde92b790b79d"} Nov 02 23:11:04.697286 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:04.697084 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f78cc76224f15261c9f4a542155459afca9cc428689c5e24db1cde92b790b79d" Nov 02 23:11:04.816209 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:04.816174 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:11:04.830189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:04.830132 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:11:05.076001 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:05.075961 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:11:05.715305 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:05.715274 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:11:06.168338 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:06.168290 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz"] Nov 02 23:11:15.769333 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.769296 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw"] Nov 02 23:11:15.769747 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.769733 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a94e21cd-1b1a-4972-8d47-18fb8e3c0856" containerName="revoke-keys" Nov 02 23:11:15.769791 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.769747 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94e21cd-1b1a-4972-8d47-18fb8e3c0856" containerName="revoke-keys" Nov 02 23:11:15.769844 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.769835 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94e21cd-1b1a-4972-8d47-18fb8e3c0856" containerName="revoke-keys" Nov 02 23:11:15.773661 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.773632 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:15.826741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.826695 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw"] Nov 02 23:11:15.869300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.869268 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:11:15.869572 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.869297 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:11:15.869700 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.869333 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:11:15.869700 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.869378 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:11:15.910431 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.910394 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b67ff28-nwvtw\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:15.910605 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.910542 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcwvb\" (UniqueName: \"kubernetes.io/projected/5363a3ce-2701-480c-a400-637ad1c33e86-kube-api-access-fcwvb\") pod \"maas-api-revoke-sub-3b67ff28-nwvtw\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:15.910647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:15.910603 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b67ff28-nwvtw\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:16.011439 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:16.011407 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcwvb\" (UniqueName: \"kubernetes.io/projected/5363a3ce-2701-480c-a400-637ad1c33e86-kube-api-access-fcwvb\") pod \"maas-api-revoke-sub-3b67ff28-nwvtw\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:16.011615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:16.011486 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b67ff28-nwvtw\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:16.011615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:16.011584 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b67ff28-nwvtw\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:16.012244 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:16.012221 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b67ff28-nwvtw\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:16.013850 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:16.013826 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b67ff28-nwvtw\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:16.064745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:16.064673 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcwvb\" (UniqueName: \"kubernetes.io/projected/5363a3ce-2701-480c-a400-637ad1c33e86-kube-api-access-fcwvb\") pod \"maas-api-revoke-sub-3b67ff28-nwvtw\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:16.089576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:16.089538 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:16.459415 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:16.459381 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw"] Nov 02 23:11:16.460565 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:11:16.460530 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5363a3ce_2701_480c_a400_637ad1c33e86.slice/crio-1a12b41693fd31fbdf943c2aee6b5da0f0a9773d4a15c369f83780a022fe0a3d WatchSource:0}: Error finding container 1a12b41693fd31fbdf943c2aee6b5da0f0a9773d4a15c369f83780a022fe0a3d: Status 404 returned error can't find the container with id 1a12b41693fd31fbdf943c2aee6b5da0f0a9773d4a15c369f83780a022fe0a3d Nov 02 23:11:16.785906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:16.785860 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" event={"ID":"5363a3ce-2701-480c-a400-637ad1c33e86","Type":"ContainerStarted","Data":"cec755b40ea9dd935718eaee65fd7fb18b66e02a188bb13bd23a81f3d63ae558"} Nov 02 23:11:16.786355 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:16.785920 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" event={"ID":"5363a3ce-2701-480c-a400-637ad1c33e86","Type":"ContainerStarted","Data":"1a12b41693fd31fbdf943c2aee6b5da0f0a9773d4a15c369f83780a022fe0a3d"} Nov 02 23:11:16.893342 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:16.893283 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" podStartSLOduration=1.8932682280000002 podStartE2EDuration="1.893268228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:16.890847839 +0000 UTC m=+1070.298159669" watchObservedRunningTime="2026-11-02 23:11:16.893268228 +0000 UTC m=+1070.300580058" Nov 02 23:11:17.793881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:17.793845 5672 generic.go:356] "Generic (PLEG): container finished" podID="5363a3ce-2701-480c-a400-637ad1c33e86" containerID="cec755b40ea9dd935718eaee65fd7fb18b66e02a188bb13bd23a81f3d63ae558" exitCode=0 Nov 02 23:11:17.794351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:17.793933 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" event={"ID":"5363a3ce-2701-480c-a400-637ad1c33e86","Type":"ContainerDied","Data":"cec755b40ea9dd935718eaee65fd7fb18b66e02a188bb13bd23a81f3d63ae558"} Nov 02 23:11:18.964261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:18.964234 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:19.045298 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.045260 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-openshift-ca\") pod \"5363a3ce-2701-480c-a400-637ad1c33e86\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " Nov 02 23:11:19.045298 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.045311 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-xks-ca\") pod \"5363a3ce-2701-480c-a400-637ad1c33e86\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " Nov 02 23:11:19.045645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.045541 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5363a3ce-2701-480c-a400-637ad1c33e86-kube-api-access-fcwvb\" (UniqueName: \"kubernetes.io/projected/5363a3ce-2701-480c-a400-637ad1c33e86-kube-api-access-fcwvb\") pod \"5363a3ce-2701-480c-a400-637ad1c33e86\" (UID: \"5363a3ce-2701-480c-a400-637ad1c33e86\") " Nov 02 23:11:19.045745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.045716 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-openshift-ca" pod "5363a3ce-2701-480c-a400-637ad1c33e86" (UID: "5363a3ce-2701-480c-a400-637ad1c33e86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:11:19.045991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.045972 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:19.047659 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.047635 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5363a3ce-2701-480c-a400-637ad1c33e86-kube-api-access-fcwvb" pod "5363a3ce-2701-480c-a400-637ad1c33e86" (UID: "5363a3ce-2701-480c-a400-637ad1c33e86"). InnerVolumeSpecName "kube-api-access-fcwvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:11:19.047821 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.047808 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-xks-ca" pod "5363a3ce-2701-480c-a400-637ad1c33e86" (UID: "5363a3ce-2701-480c-a400-637ad1c33e86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:11:19.147567 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.147531 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcwvb\" (UniqueName: \"kubernetes.io/projected/5363a3ce-2701-480c-a400-637ad1c33e86-kube-api-access-fcwvb\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:19.147567 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.147561 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5363a3ce-2701-480c-a400-637ad1c33e86-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:19.807633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.807607 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" Nov 02 23:11:19.807787 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.807603 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw" event={"ID":"5363a3ce-2701-480c-a400-637ad1c33e86","Type":"ContainerDied","Data":"1a12b41693fd31fbdf943c2aee6b5da0f0a9773d4a15c369f83780a022fe0a3d"} Nov 02 23:11:19.807787 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:19.807729 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a12b41693fd31fbdf943c2aee6b5da0f0a9773d4a15c369f83780a022fe0a3d" Nov 02 23:11:21.209087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.209043 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" podUID="fdba3b13-1172-4ec7-bb34-b882a753f6f6" containerName="main" containerID="cri-o://0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5" gracePeriod=45 Nov 02 23:11:21.491596 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.491569 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:11:21.574196 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574165 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tmp-dir\") pod \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " Nov 02 23:11:21.574400 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574281 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-dshm\") pod \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " Nov 02 23:11:21.574400 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574329 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-home\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-home\") pod \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " Nov 02 23:11:21.574400 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574364 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-model-cache\") pod \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " Nov 02 23:11:21.574576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574441 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdba3b13-1172-4ec7-bb34-b882a753f6f6-kube-api-access-6zjqz\" (UniqueName: \"kubernetes.io/projected/fdba3b13-1172-4ec7-bb34-b882a753f6f6-kube-api-access-6zjqz\") pod \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " Nov 02 23:11:21.574576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574476 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tls-certs\") pod \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\" (UID: \"fdba3b13-1172-4ec7-bb34-b882a753f6f6\") " Nov 02 23:11:21.574576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574501 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tmp-dir" pod "fdba3b13-1172-4ec7-bb34-b882a753f6f6" (UID: "fdba3b13-1172-4ec7-bb34-b882a753f6f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:11:21.574701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574586 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-home" pod "fdba3b13-1172-4ec7-bb34-b882a753f6f6" (UID: "fdba3b13-1172-4ec7-bb34-b882a753f6f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:11:21.574701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574603 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-model-cache" pod "fdba3b13-1172-4ec7-bb34-b882a753f6f6" (UID: "fdba3b13-1172-4ec7-bb34-b882a753f6f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:11:21.574879 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574857 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:21.574879 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574878 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:21.575037 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.574891 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:21.576626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.576594 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-dshm" pod "fdba3b13-1172-4ec7-bb34-b882a753f6f6" (UID: "fdba3b13-1172-4ec7-bb34-b882a753f6f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:11:21.576710 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.576645 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tls-certs" pod "fdba3b13-1172-4ec7-bb34-b882a753f6f6" (UID: "fdba3b13-1172-4ec7-bb34-b882a753f6f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:11:21.576710 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.576648 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdba3b13-1172-4ec7-bb34-b882a753f6f6-kube-api-access-6zjqz" pod "fdba3b13-1172-4ec7-bb34-b882a753f6f6" (UID: "fdba3b13-1172-4ec7-bb34-b882a753f6f6"). InnerVolumeSpecName "kube-api-access-6zjqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:11:21.676054 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.676012 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdba3b13-1172-4ec7-bb34-b882a753f6f6-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:21.676054 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.676048 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdba3b13-1172-4ec7-bb34-b882a753f6f6-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:21.676054 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.676058 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zjqz\" (UniqueName: \"kubernetes.io/projected/fdba3b13-1172-4ec7-bb34-b882a753f6f6-kube-api-access-6zjqz\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:21.822078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.822039 5672 generic.go:356] "Generic (PLEG): container finished" podID="fdba3b13-1172-4ec7-bb34-b882a753f6f6" containerID="0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5" exitCode=0 Nov 02 23:11:21.822078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.822090 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" Nov 02 23:11:21.822362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.822111 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" event={"ID":"fdba3b13-1172-4ec7-bb34-b882a753f6f6","Type":"ContainerDied","Data":"0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5"} Nov 02 23:11:21.822362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.822164 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz" event={"ID":"fdba3b13-1172-4ec7-bb34-b882a753f6f6","Type":"ContainerDied","Data":"49728a20be63201dde53bfe5db1b5fcb18e7cfd479dfb4010cb6217110e8a220"} Nov 02 23:11:21.822362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.822190 5672 scope.go:122] "RemoveContainer" containerID="0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5" Nov 02 23:11:21.834000 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.833972 5672 scope.go:122] "RemoveContainer" containerID="0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5" Nov 02 23:11:21.834389 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:11:21.834361 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5\": container with ID starting with 0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5 not found: ID does not exist" containerID="0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5" Nov 02 23:11:21.834472 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.834401 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5"} err="failed to get container status \"0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5\": rpc error: code = NotFound desc = could not find container \"0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5\": container with ID starting with 0e5d3a5846595da51e651736c9a818fc823519c19bff87981f349ef6c85569b5 not found: ID does not exist" Nov 02 23:11:21.981381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.981346 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz"] Nov 02 23:11:21.985289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:21.985263 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-a-11bf0f82-kserve-6b8c84c8f-gz6dz"] Nov 02 23:11:22.712611 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:22.712569 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdba3b13-1172-4ec7-bb34-b882a753f6f6" path="/var/lib/kubelet/pods/fdba3b13-1172-4ec7-bb34-b882a753f6f6/volumes" Nov 02 23:11:26.049045 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:11:26.046730 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:11:26.704016 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.703979 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw"] Nov 02 23:11:26.704711 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.704690 5672 cpu_manager.go:414] "containerMap: removing container" podUID="fdba3b13-1172-4ec7-bb34-b882a753f6f6" containerName="main" Nov 02 23:11:26.704800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.704713 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdba3b13-1172-4ec7-bb34-b882a753f6f6" containerName="main" Nov 02 23:11:26.704800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.704739 5672 cpu_manager.go:414] "containerMap: removing container" podUID="5363a3ce-2701-480c-a400-637ad1c33e86" containerName="revoke-keys" Nov 02 23:11:26.704800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.704747 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5363a3ce-2701-480c-a400-637ad1c33e86" containerName="revoke-keys" Nov 02 23:11:26.704893 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.704839 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdba3b13-1172-4ec7-bb34-b882a753f6f6" containerName="main" Nov 02 23:11:26.704893 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.704851 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="5363a3ce-2701-480c-a400-637ad1c33e86" containerName="revoke-keys" Nov 02 23:11:26.709554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.709530 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:26.749818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.749780 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17ca43d2-v7hzw\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:26.750018 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.749829 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slszt\" (UniqueName: \"kubernetes.io/projected/365284f7-cfb6-4a3f-a02d-b69db1a44e15-kube-api-access-slszt\") pod \"maas-api-revoke-sub-17ca43d2-v7hzw\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:26.750018 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.749986 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17ca43d2-v7hzw\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:26.761380 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.761345 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:11:26.761633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.761611 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:11:26.761989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.761972 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:11:26.808351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.808318 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:11:26.812882 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.812852 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw"] Nov 02 23:11:26.850911 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.850879 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17ca43d2-v7hzw\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:26.851122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.850948 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17ca43d2-v7hzw\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:26.851122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.850992 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slszt\" (UniqueName: \"kubernetes.io/projected/365284f7-cfb6-4a3f-a02d-b69db1a44e15-kube-api-access-slszt\") pod \"maas-api-revoke-sub-17ca43d2-v7hzw\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:26.851645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.851619 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17ca43d2-v7hzw\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:26.853628 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.853603 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17ca43d2-v7hzw\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:26.915838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.915798 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slszt\" (UniqueName: \"kubernetes.io/projected/365284f7-cfb6-4a3f-a02d-b69db1a44e15-kube-api-access-slszt\") pod \"maas-api-revoke-sub-17ca43d2-v7hzw\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:26.993638 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.993598 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b"] Nov 02 23:11:26.998626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:26.998601 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.024162 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.024127 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:27.070597 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.070424 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b"] Nov 02 23:11:27.154868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.154182 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.154868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.154248 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.154868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.154336 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.154868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.154381 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.154868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.154419 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.154868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.154504 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.154868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.154540 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.154868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.154567 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94vl6\" (UniqueName: \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-kube-api-access-94vl6\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.154868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.154613 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.256267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.256166 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.256267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.256232 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.256495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.256315 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.256495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.256356 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.256495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.256395 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.256495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.256479 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.256699 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.256515 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.256699 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.256539 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vl6\" (UniqueName: \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-kube-api-access-94vl6\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.256699 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.256583 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.256699 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.256632 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.257029 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.256999 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.257165 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.257010 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.257242 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.257187 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.258121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.258097 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.259293 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.259270 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.259581 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.259561 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.310598 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.310556 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.310783 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.310665 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vl6\" (UniqueName: \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-kube-api-access-94vl6\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.313390 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.313367 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:27.412131 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.412027 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw"] Nov 02 23:11:27.413925 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:11:27.413894 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365284f7_cfb6_4a3f_a02d_b69db1a44e15.slice/crio-5b1c3831ca7fc091d1807aef8023df78bfea06c27c515e0731e5c44ef43156da WatchSource:0}: Error finding container 5b1c3831ca7fc091d1807aef8023df78bfea06c27c515e0731e5c44ef43156da: Status 404 returned error can't find the container with id 5b1c3831ca7fc091d1807aef8023df78bfea06c27c515e0731e5c44ef43156da Nov 02 23:11:27.686611 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.686586 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b"] Nov 02 23:11:27.726439 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:11:27.726402 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7211e4c9_9a0a_4cb4_ac8d_9ab2ee2643bb.slice/crio-66d7715530a25a432eb6ca725cf62d3130e92999cadb63113fca957a00250faf WatchSource:0}: Error finding container 66d7715530a25a432eb6ca725cf62d3130e92999cadb63113fca957a00250faf: Status 404 returned error can't find the container with id 66d7715530a25a432eb6ca725cf62d3130e92999cadb63113fca957a00250faf Nov 02 23:11:27.729174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.729126 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:27.729268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.729217 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:27.729268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.729249 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:27.867610 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.867570 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" event={"ID":"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb","Type":"ContainerStarted","Data":"b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb"} Nov 02 23:11:27.867807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.867643 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" event={"ID":"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb","Type":"ContainerStarted","Data":"66d7715530a25a432eb6ca725cf62d3130e92999cadb63113fca957a00250faf"} Nov 02 23:11:27.869465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.869432 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" event={"ID":"365284f7-cfb6-4a3f-a02d-b69db1a44e15","Type":"ContainerStarted","Data":"103000fa8c498c2bdc3e21150c77ba47f0dac6fe33882bc83c7864315477bf98"} Nov 02 23:11:27.869599 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.869481 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" event={"ID":"365284f7-cfb6-4a3f-a02d-b69db1a44e15","Type":"ContainerStarted","Data":"5b1c3831ca7fc091d1807aef8023df78bfea06c27c515e0731e5c44ef43156da"} Nov 02 23:11:27.986261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:27.985727 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" podStartSLOduration=1.985708397 podStartE2EDuration="1.985708397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:27.979346843 +0000 UTC m=+1081.386658674" watchObservedRunningTime="2026-11-02 23:11:27.985708397 +0000 UTC m=+1081.393020229" Nov 02 23:11:28.089962 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.089909 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" podStartSLOduration=2.08989534 podStartE2EDuration="2.08989534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:28.086910178 +0000 UTC m=+1081.494222008" watchObservedRunningTime="2026-11-02 23:11:28.08989534 +0000 UTC m=+1081.497207170" Nov 02 23:11:28.261228 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.260285 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg"] Nov 02 23:11:28.264775 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.264749 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.313801 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.313769 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:28.315391 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.315364 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:11:28.315509 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.315413 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:11:28.333395 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.331429 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3d8404-openshift-default-dockercfg-kh456\"" Nov 02 23:11:28.355173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.347810 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg"] Nov 02 23:11:28.374397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.374339 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.374647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.374406 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.374647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.374454 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.374647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.374490 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18cd3f1-549c-4fee-9cfd-f6844650c560-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.374647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.374528 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.374647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.374554 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.374647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.374618 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.374647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.374645 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkz2\" (UniqueName: \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-kube-api-access-mtkz2\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.375082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.374735 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.475298 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.475265 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.475466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.475321 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.475466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.475358 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.475466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.475392 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.475466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.475432 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18cd3f1-549c-4fee-9cfd-f6844650c560-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.475647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.475470 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.475647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.475495 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.475647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.475564 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.475647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.475587 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkz2\" (UniqueName: \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-kube-api-access-mtkz2\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.475835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.475806 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.476014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.475990 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.476096 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.476008 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.476096 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.476052 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.476408 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.476363 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18cd3f1-549c-4fee-9cfd-f6844650c560-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.477873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.477851 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.478179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.478127 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.546227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.536469 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.552536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.552493 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkz2\" (UniqueName: \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-kube-api-access-mtkz2\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.581331 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.581298 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:28.877357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.877276 5672 generic.go:356] "Generic (PLEG): container finished" podID="365284f7-cfb6-4a3f-a02d-b69db1a44e15" containerID="103000fa8c498c2bdc3e21150c77ba47f0dac6fe33882bc83c7864315477bf98" exitCode=0 Nov 02 23:11:28.877498 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:28.877356 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" event={"ID":"365284f7-cfb6-4a3f-a02d-b69db1a44e15","Type":"ContainerDied","Data":"103000fa8c498c2bdc3e21150c77ba47f0dac6fe33882bc83c7864315477bf98"} Nov 02 23:11:29.059648 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:29.059605 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg"] Nov 02 23:11:29.064200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:29.064168 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:29.064295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:29.064236 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:29.064295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:29.064266 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:29.314074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:29.314040 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:11:29.314515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:29.314096 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:11:29.885807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:29.885772 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" event={"ID":"d18cd3f1-549c-4fee-9cfd-f6844650c560","Type":"ContainerStarted","Data":"9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee"} Nov 02 23:11:29.886015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:29.885824 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" event={"ID":"d18cd3f1-549c-4fee-9cfd-f6844650c560","Type":"ContainerStarted","Data":"7664ad3e88f9e761f3951eb28ff899f2d90502819893dfaf87eeb1cf2313f5df"} Nov 02 23:11:29.998048 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:29.998003 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" podStartSLOduration=1.997987897 podStartE2EDuration="1.997987897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:29.997457664 +0000 UTC m=+1083.404769537" watchObservedRunningTime="2026-11-02 23:11:29.997987897 +0000 UTC m=+1083.405299726" Nov 02 23:11:30.104527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.104498 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:30.193212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.193065 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365284f7-cfb6-4a3f-a02d-b69db1a44e15-kube-api-access-slszt\" (UniqueName: \"kubernetes.io/projected/365284f7-cfb6-4a3f-a02d-b69db1a44e15-kube-api-access-slszt\") pod \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " Nov 02 23:11:30.193212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.193206 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-xks-ca\") pod \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " Nov 02 23:11:30.193470 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.193264 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-openshift-ca\") pod \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\" (UID: \"365284f7-cfb6-4a3f-a02d-b69db1a44e15\") " Nov 02 23:11:30.193711 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.193686 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-openshift-ca" pod "365284f7-cfb6-4a3f-a02d-b69db1a44e15" (UID: "365284f7-cfb6-4a3f-a02d-b69db1a44e15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:11:30.195486 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.195462 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-xks-ca" pod "365284f7-cfb6-4a3f-a02d-b69db1a44e15" (UID: "365284f7-cfb6-4a3f-a02d-b69db1a44e15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:11:30.195556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.195466 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365284f7-cfb6-4a3f-a02d-b69db1a44e15-kube-api-access-slszt" pod "365284f7-cfb6-4a3f-a02d-b69db1a44e15" (UID: "365284f7-cfb6-4a3f-a02d-b69db1a44e15"). InnerVolumeSpecName "kube-api-access-slszt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:11:30.294499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.294458 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slszt\" (UniqueName: \"kubernetes.io/projected/365284f7-cfb6-4a3f-a02d-b69db1a44e15-kube-api-access-slszt\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:30.294499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.294502 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:30.294678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.294518 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365284f7-cfb6-4a3f-a02d-b69db1a44e15-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:30.313944 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.313914 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:11:30.314102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.313964 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:11:30.581484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.581445 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:30.582812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.582789 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:11:30.582871 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.582838 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 23:11:30.894910 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.894830 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" Nov 02 23:11:30.898375 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.898332 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw" event={"ID":"365284f7-cfb6-4a3f-a02d-b69db1a44e15","Type":"ContainerDied","Data":"5b1c3831ca7fc091d1807aef8023df78bfea06c27c515e0731e5c44ef43156da"} Nov 02 23:11:30.898489 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:30.898411 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b1c3831ca7fc091d1807aef8023df78bfea06c27c515e0731e5c44ef43156da" Nov 02 23:11:31.106257 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.106221 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd"] Nov 02 23:11:31.106730 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.106713 5672 cpu_manager.go:414] "containerMap: removing container" podUID="365284f7-cfb6-4a3f-a02d-b69db1a44e15" containerName="revoke-keys" Nov 02 23:11:31.106730 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.106730 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365284f7-cfb6-4a3f-a02d-b69db1a44e15" containerName="revoke-keys" Nov 02 23:11:31.106866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.106853 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="365284f7-cfb6-4a3f-a02d-b69db1a44e15" containerName="revoke-keys" Nov 02 23:11:31.111013 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.110993 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.211098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.210794 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z22f8\" (UniqueName: \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-kube-api-access-z22f8\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.211098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.210846 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.211098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.210895 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.211098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.210945 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.211098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.211074 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.211547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.211175 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.211547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.211210 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.211547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.211267 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.211547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.211317 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.215546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.215514 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd"] Nov 02 23:11:31.312721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.312685 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z22f8\" (UniqueName: \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-kube-api-access-z22f8\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.312721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.312726 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.312932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.312757 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.312932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.312789 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.312932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.312819 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.312932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.312840 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.312932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.312859 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.312932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.312896 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.313109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.312932 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.313519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.313496 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.313519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.313510 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.314003 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.313978 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.314255 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.313988 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.314646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.314284 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.314646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.314291 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:11:31.314646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.314403 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:11:31.315478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.315441 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.317121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.317096 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.381946 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.381910 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z22f8\" (UniqueName: \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-kube-api-access-z22f8\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.386194 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.385803 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.428175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.424273 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:31.586012 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.585574 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:11:31.586012 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.585638 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 23:11:31.976479 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.976429 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd"] Nov 02 23:11:31.985196 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:11:31.985118 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799e0d2a_5dce_470c_a73d_dd9b27258dd2.slice/crio-8d9f7098a2aee5efb7cd638b744c12b8728b9c5aa6543cdb91cdb2e20ad52c8d WatchSource:0}: Error finding container 8d9f7098a2aee5efb7cd638b744c12b8728b9c5aa6543cdb91cdb2e20ad52c8d: Status 404 returned error can't find the container with id 8d9f7098a2aee5efb7cd638b744c12b8728b9c5aa6543cdb91cdb2e20ad52c8d Nov 02 23:11:31.988692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.988399 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:31.988692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.988489 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:31.988692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:31.988533 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:11:32.314434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:32.314330 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:11:32.315135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:32.315102 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:11:32.583227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:32.582641 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:11:32.583227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:32.582717 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 23:11:32.980520 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:32.976297 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" event={"ID":"799e0d2a-5dce-470c-a73d-dd9b27258dd2","Type":"ContainerStarted","Data":"d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb"} Nov 02 23:11:32.980520 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:32.976369 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" event={"ID":"799e0d2a-5dce-470c-a73d-dd9b27258dd2","Type":"ContainerStarted","Data":"8d9f7098a2aee5efb7cd638b744c12b8728b9c5aa6543cdb91cdb2e20ad52c8d"} Nov 02 23:11:33.127174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:33.126383 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" podStartSLOduration=2.126363553 podStartE2EDuration="2.126363553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:33.104578736 +0000 UTC m=+1086.511890567" watchObservedRunningTime="2026-11-02 23:11:33.126363553 +0000 UTC m=+1086.533675385" Nov 02 23:11:33.314690 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:33.314592 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:11:33.314690 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:33.314647 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:11:33.432863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:33.432832 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:33.449240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:33.449185 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 23:11:33.449541 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:33.449509 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" podUID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 23:11:33.585437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:33.584918 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:11:33.585437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:33.584982 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 23:11:34.314098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:34.314061 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 23:11:34.314317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:34.314110 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 23:11:34.433172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:34.430694 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 23:11:34.433172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:34.430749 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" podUID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 23:11:34.582658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:34.582563 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:11:34.583074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:34.583032 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 23:11:35.104315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:35.104228 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt"] Nov 02 23:11:35.110546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:35.110511 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872d6677-hlkqt"] Nov 02 23:11:35.429233 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:35.428801 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 23:11:35.429233 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:35.428860 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" podUID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 23:11:35.582652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:35.582556 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 23:11:35.582652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:35.582636 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 23:11:35.668996 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:35.668966 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:35.669721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:35.669693 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:35.670813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:35.670794 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:11:36.439618 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:36.439581 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 23:11:36.439903 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:36.439876 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" podUID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 23:11:36.713396 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:36.713314 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e898c01-bb37-4fba-867b-e67b65862162" path="/var/lib/kubelet/pods/7e898c01-bb37-4fba-867b-e67b65862162/volumes" Nov 02 23:11:36.946515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:36.946482 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:36.948206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:36.948184 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:36.963330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:36.963182 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:11:37.431520 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:37.431488 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 23:11:37.431796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:37.431767 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" podUID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 23:11:38.428738 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:38.428700 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 23:11:38.429399 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:38.429357 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" podUID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 23:11:39.136849 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:39.136818 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:11:39.433617 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:39.433537 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:39.434011 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:39.433948 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:39.434759 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:39.434738 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:11:42.820638 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:42.820605 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2"] Nov 02 23:11:42.823955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:42.823936 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:42.873100 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:42.873069 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:11:42.887650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:42.887625 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:11:42.892434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:42.892407 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2"] Nov 02 23:11:42.919779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:42.919759 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:11:42.919779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:42.919775 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:11:42.959690 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:42.959663 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6ff5280-gjmk2\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:42.959822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:42.959705 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6ff5280-gjmk2\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:42.959889 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:42.959850 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64cdr\" (UniqueName: \"kubernetes.io/projected/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-kube-api-access-64cdr\") pod \"maas-api-revoke-sub-b6ff5280-gjmk2\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:43.060786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:43.060750 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cdr\" (UniqueName: \"kubernetes.io/projected/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-kube-api-access-64cdr\") pod \"maas-api-revoke-sub-b6ff5280-gjmk2\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:43.060969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:43.060839 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6ff5280-gjmk2\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:43.060969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:43.060872 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6ff5280-gjmk2\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:43.061574 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:43.061552 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6ff5280-gjmk2\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:43.063135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:43.063109 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6ff5280-gjmk2\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:43.114461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:43.114390 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64cdr\" (UniqueName: \"kubernetes.io/projected/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-kube-api-access-64cdr\") pod \"maas-api-revoke-sub-b6ff5280-gjmk2\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:43.137728 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:43.137703 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:43.515350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:43.515326 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2"] Nov 02 23:11:43.517190 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:11:43.517144 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f0f9b5_393e_49c5_a002_e7f6117cdcf5.slice/crio-96354c553590fdd6734ea2c8d32ff0220cea2d055c3a848aa14644657f66a9da WatchSource:0}: Error finding container 96354c553590fdd6734ea2c8d32ff0220cea2d055c3a848aa14644657f66a9da: Status 404 returned error can't find the container with id 96354c553590fdd6734ea2c8d32ff0220cea2d055c3a848aa14644657f66a9da Nov 02 23:11:44.076824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:44.076785 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" event={"ID":"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5","Type":"ContainerStarted","Data":"98704f72d8fa163d6a5aa442186e940899902b43c015c030470c42c4a0b96e60"} Nov 02 23:11:44.077197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:44.076843 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" event={"ID":"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5","Type":"ContainerStarted","Data":"96354c553590fdd6734ea2c8d32ff0220cea2d055c3a848aa14644657f66a9da"} Nov 02 23:11:44.185013 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:44.184921 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" podStartSLOduration=2.184907893 podStartE2EDuration="2.184907893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:44.183142052 +0000 UTC m=+1097.590453882" watchObservedRunningTime="2026-11-02 23:11:44.184907893 +0000 UTC m=+1097.592219723" Nov 02 23:11:45.083892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:45.083856 5672 generic.go:356] "Generic (PLEG): container finished" podID="c4f0f9b5-393e-49c5-a002-e7f6117cdcf5" containerID="98704f72d8fa163d6a5aa442186e940899902b43c015c030470c42c4a0b96e60" exitCode=0 Nov 02 23:11:45.084350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:45.083946 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" event={"ID":"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5","Type":"ContainerDied","Data":"98704f72d8fa163d6a5aa442186e940899902b43c015c030470c42c4a0b96e60"} Nov 02 23:11:46.230340 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:46.230313 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:46.297633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:46.297600 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-openshift-ca\") pod \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " Nov 02 23:11:46.297784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:46.297655 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-kube-api-access-64cdr\" (UniqueName: \"kubernetes.io/projected/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-kube-api-access-64cdr\") pod \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " Nov 02 23:11:46.297854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:46.297840 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-xks-ca\") pod \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\" (UID: \"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5\") " Nov 02 23:11:46.298038 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:46.298009 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-openshift-ca" pod "c4f0f9b5-393e-49c5-a002-e7f6117cdcf5" (UID: "c4f0f9b5-393e-49c5-a002-e7f6117cdcf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:11:46.298212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:46.298194 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:46.299781 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:46.299755 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-kube-api-access-64cdr" pod "c4f0f9b5-393e-49c5-a002-e7f6117cdcf5" (UID: "c4f0f9b5-393e-49c5-a002-e7f6117cdcf5"). InnerVolumeSpecName "kube-api-access-64cdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:11:46.299912 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:46.299885 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-xks-ca" pod "c4f0f9b5-393e-49c5-a002-e7f6117cdcf5" (UID: "c4f0f9b5-393e-49c5-a002-e7f6117cdcf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:11:46.399583 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:46.399510 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64cdr\" (UniqueName: \"kubernetes.io/projected/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-kube-api-access-64cdr\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:46.399583 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:46.399536 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:47.098196 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:47.098112 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" Nov 02 23:11:47.098196 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:47.098134 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2" event={"ID":"c4f0f9b5-393e-49c5-a002-e7f6117cdcf5","Type":"ContainerDied","Data":"96354c553590fdd6734ea2c8d32ff0220cea2d055c3a848aa14644657f66a9da"} Nov 02 23:11:47.098196 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:47.098192 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96354c553590fdd6734ea2c8d32ff0220cea2d055c3a848aa14644657f66a9da" Nov 02 23:11:52.764286 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.764246 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr"] Nov 02 23:11:52.764978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.764946 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c4f0f9b5-393e-49c5-a002-e7f6117cdcf5" containerName="revoke-keys" Nov 02 23:11:52.764978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.764974 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f0f9b5-393e-49c5-a002-e7f6117cdcf5" containerName="revoke-keys" Nov 02 23:11:52.765133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.765124 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f0f9b5-393e-49c5-a002-e7f6117cdcf5" containerName="revoke-keys" Nov 02 23:11:52.768841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.768813 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:52.818315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.818279 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:11:52.864690 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.864660 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:11:52.864997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.864980 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:11:52.865222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.865206 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:11:52.869618 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.869588 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr"] Nov 02 23:11:52.875228 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.875200 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkfqj\" (UniqueName: \"kubernetes.io/projected/404d1f5c-dc02-480f-ae33-7c8603778e33-kube-api-access-fkfqj\") pod \"maas-api-revoke-sub-54a42de9-5tqcr\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:52.875340 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.875243 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a42de9-5tqcr\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:52.875412 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.875348 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a42de9-5tqcr\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:52.976975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.976941 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkfqj\" (UniqueName: \"kubernetes.io/projected/404d1f5c-dc02-480f-ae33-7c8603778e33-kube-api-access-fkfqj\") pod \"maas-api-revoke-sub-54a42de9-5tqcr\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:52.977178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.976989 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a42de9-5tqcr\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:52.977178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.977077 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a42de9-5tqcr\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:52.977747 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.977721 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a42de9-5tqcr\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:52.979251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:52.979227 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a42de9-5tqcr\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:53.030536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:53.030483 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkfqj\" (UniqueName: \"kubernetes.io/projected/404d1f5c-dc02-480f-ae33-7c8603778e33-kube-api-access-fkfqj\") pod \"maas-api-revoke-sub-54a42de9-5tqcr\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:53.044909 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:53.044886 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc"] Nov 02 23:11:53.047903 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:53.047881 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e3ef70-4fqdc"] Nov 02 23:11:53.085569 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:53.085548 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:53.453797 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:53.453766 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr"] Nov 02 23:11:53.456514 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:11:53.456473 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404d1f5c_dc02_480f_ae33_7c8603778e33.slice/crio-dac017689c3422b7e40deabde7fab5762c2fbfd959bcdfb9d76eccda691ec513 WatchSource:0}: Error finding container dac017689c3422b7e40deabde7fab5762c2fbfd959bcdfb9d76eccda691ec513: Status 404 returned error can't find the container with id dac017689c3422b7e40deabde7fab5762c2fbfd959bcdfb9d76eccda691ec513 Nov 02 23:11:54.153392 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:54.153283 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" event={"ID":"404d1f5c-dc02-480f-ae33-7c8603778e33","Type":"ContainerStarted","Data":"77352e0ac2ace52d3e61a93916caacbb948a7f7f0a91e9735c19ef89ca0525e1"} Nov 02 23:11:54.153856 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:54.153392 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" event={"ID":"404d1f5c-dc02-480f-ae33-7c8603778e33","Type":"ContainerStarted","Data":"dac017689c3422b7e40deabde7fab5762c2fbfd959bcdfb9d76eccda691ec513"} Nov 02 23:11:54.264618 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:54.264557 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" podStartSLOduration=2.26453907 podStartE2EDuration="2.26453907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:54.263046539 +0000 UTC m=+1107.670358371" watchObservedRunningTime="2026-11-02 23:11:54.26453907 +0000 UTC m=+1107.671850906" Nov 02 23:11:54.713467 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:54.713435 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2441016-6522-4f36-996e-e54ca4d73534" path="/var/lib/kubelet/pods/a2441016-6522-4f36-996e-e54ca4d73534/volumes" Nov 02 23:11:55.166906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:55.166869 5672 generic.go:356] "Generic (PLEG): container finished" podID="404d1f5c-dc02-480f-ae33-7c8603778e33" containerID="77352e0ac2ace52d3e61a93916caacbb948a7f7f0a91e9735c19ef89ca0525e1" exitCode=0 Nov 02 23:11:55.167529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:55.166939 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" event={"ID":"404d1f5c-dc02-480f-ae33-7c8603778e33","Type":"ContainerDied","Data":"77352e0ac2ace52d3e61a93916caacbb948a7f7f0a91e9735c19ef89ca0525e1"} Nov 02 23:11:56.020656 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:11:56.020618 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:11:56.322845 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:56.322810 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:56.416745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:56.416715 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404d1f5c-dc02-480f-ae33-7c8603778e33-kube-api-access-fkfqj\" (UniqueName: \"kubernetes.io/projected/404d1f5c-dc02-480f-ae33-7c8603778e33-kube-api-access-fkfqj\") pod \"404d1f5c-dc02-480f-ae33-7c8603778e33\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " Nov 02 23:11:56.416929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:56.416775 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-openshift-ca\") pod \"404d1f5c-dc02-480f-ae33-7c8603778e33\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " Nov 02 23:11:56.416929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:56.416839 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-xks-ca\") pod \"404d1f5c-dc02-480f-ae33-7c8603778e33\" (UID: \"404d1f5c-dc02-480f-ae33-7c8603778e33\") " Nov 02 23:11:56.417204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:56.417180 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-openshift-ca" pod "404d1f5c-dc02-480f-ae33-7c8603778e33" (UID: "404d1f5c-dc02-480f-ae33-7c8603778e33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:11:56.418887 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:56.418858 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404d1f5c-dc02-480f-ae33-7c8603778e33-kube-api-access-fkfqj" pod "404d1f5c-dc02-480f-ae33-7c8603778e33" (UID: "404d1f5c-dc02-480f-ae33-7c8603778e33"). InnerVolumeSpecName "kube-api-access-fkfqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:11:56.419100 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:56.419063 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-xks-ca" pod "404d1f5c-dc02-480f-ae33-7c8603778e33" (UID: "404d1f5c-dc02-480f-ae33-7c8603778e33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:11:56.518116 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:56.518078 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkfqj\" (UniqueName: \"kubernetes.io/projected/404d1f5c-dc02-480f-ae33-7c8603778e33-kube-api-access-fkfqj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:56.518116 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:56.518108 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:56.518116 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:56.518122 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404d1f5c-dc02-480f-ae33-7c8603778e33-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:11:57.180981 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:57.180879 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" Nov 02 23:11:57.180981 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:57.180899 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr" event={"ID":"404d1f5c-dc02-480f-ae33-7c8603778e33","Type":"ContainerDied","Data":"dac017689c3422b7e40deabde7fab5762c2fbfd959bcdfb9d76eccda691ec513"} Nov 02 23:11:57.181234 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:57.180987 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dac017689c3422b7e40deabde7fab5762c2fbfd959bcdfb9d76eccda691ec513" Nov 02 23:11:58.791773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.791742 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd"] Nov 02 23:11:58.792244 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.792231 5672 cpu_manager.go:414] "containerMap: removing container" podUID="404d1f5c-dc02-480f-ae33-7c8603778e33" containerName="revoke-keys" Nov 02 23:11:58.792302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.792245 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404d1f5c-dc02-480f-ae33-7c8603778e33" containerName="revoke-keys" Nov 02 23:11:58.792346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.792338 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="404d1f5c-dc02-480f-ae33-7c8603778e33" containerName="revoke-keys" Nov 02 23:11:58.795911 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.795892 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:11:58.844517 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.844486 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c47af0-tlwbd\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:11:58.844669 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.844540 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr98w\" (UniqueName: \"kubernetes.io/projected/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-kube-api-access-lr98w\") pod \"maas-api-revoke-sub-d9c47af0-tlwbd\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:11:58.844734 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.844712 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c47af0-tlwbd\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:11:58.845288 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.845264 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:11:58.891722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.891682 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:11:58.891722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.891688 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:11:58.891917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.891869 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:11:58.894849 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.894825 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd"] Nov 02 23:11:58.946407 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.946366 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c47af0-tlwbd\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:11:58.946590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.946435 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr98w\" (UniqueName: \"kubernetes.io/projected/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-kube-api-access-lr98w\") pod \"maas-api-revoke-sub-d9c47af0-tlwbd\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:11:58.946590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.946493 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c47af0-tlwbd\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:11:58.947276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.947251 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c47af0-tlwbd\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:11:58.949479 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.949454 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c47af0-tlwbd\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:11:58.999469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:58.999436 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr98w\" (UniqueName: \"kubernetes.io/projected/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-kube-api-access-lr98w\") pod \"maas-api-revoke-sub-d9c47af0-tlwbd\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:11:59.109736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.109661 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:11:59.481259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.481228 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd"] Nov 02 23:11:59.482182 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:11:59.482132 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd1ad31_73cb_4aeb_908e_a0ea39a628b5.slice/crio-166b9a47508f1371d44dc5e3ed97bb7819844f76f4f79f8091833394e4a0149f WatchSource:0}: Error finding container 166b9a47508f1371d44dc5e3ed97bb7819844f76f4f79f8091833394e4a0149f: Status 404 returned error can't find the container with id 166b9a47508f1371d44dc5e3ed97bb7819844f76f4f79f8091833394e4a0149f Nov 02 23:11:59.491971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.491940 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65"] Nov 02 23:11:59.497335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.497218 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.546221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.546191 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"default-dockercfg-66dv6\"" Nov 02 23:11:59.546907 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.546880 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"kube-root-ca.crt\"" Nov 02 23:11:59.553460 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.553436 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97473a88-1d49-4873-970e-914e29a6908f-tls-certs\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.553605 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.553586 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-model-cache\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.553680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.553640 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-tmp-dir\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.553743 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.553698 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkwz2\" (UniqueName: \"kubernetes.io/projected/97473a88-1d49-4873-970e-914e29a6908f-kube-api-access-rkwz2\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.553807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.553747 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-home\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.553807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.553763 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-dshm\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.566457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.565275 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"distinct-w4-fb6e1d-kserve-self-signed-certs\"" Nov 02 23:11:59.566457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.565276 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"openshift-service-ca.crt\"" Nov 02 23:11:59.583362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.583338 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65"] Nov 02 23:11:59.655190 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.655136 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-model-cache\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.655369 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.655233 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-tmp-dir\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.655369 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.655269 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkwz2\" (UniqueName: \"kubernetes.io/projected/97473a88-1d49-4873-970e-914e29a6908f-kube-api-access-rkwz2\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.655369 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.655329 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-home\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.655369 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.655353 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-dshm\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.655544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.655418 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97473a88-1d49-4873-970e-914e29a6908f-tls-certs\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.657141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.656228 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-model-cache\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.657141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.656650 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-home\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.660094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.659161 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-tmp-dir\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.660238 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.660141 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97473a88-1d49-4873-970e-914e29a6908f-tls-certs\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.660384 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.660344 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-dshm\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.709650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.709611 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkwz2\" (UniqueName: \"kubernetes.io/projected/97473a88-1d49-4873-970e-914e29a6908f-kube-api-access-rkwz2\") pod \"distinct-w4-fb6e1d-kserve-5666945557-h6f65\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:11:59.876191 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:11:59.872415 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:12:00.203733 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:00.203643 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" event={"ID":"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5","Type":"ContainerStarted","Data":"33b4254f27728124cb73d44c0fad01d76b7d42696493f77ac79411e52480370c"} Nov 02 23:12:00.203733 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:00.203711 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" event={"ID":"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5","Type":"ContainerStarted","Data":"166b9a47508f1371d44dc5e3ed97bb7819844f76f4f79f8091833394e4a0149f"} Nov 02 23:12:00.294327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:00.294286 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65"] Nov 02 23:12:00.295759 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:12:00.295722 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97473a88_1d49_4873_970e_914e29a6908f.slice/crio-bdb3070af0b31ecb473259c85eba1d36021de5588870944dccd108e2f616b870 WatchSource:0}: Error finding container bdb3070af0b31ecb473259c85eba1d36021de5588870944dccd108e2f616b870: Status 404 returned error can't find the container with id bdb3070af0b31ecb473259c85eba1d36021de5588870944dccd108e2f616b870 Nov 02 23:12:00.321125 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:00.321070 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" podStartSLOduration=2.321053799 podStartE2EDuration="2.321053799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:00.317646725 +0000 UTC m=+1113.724958555" watchObservedRunningTime="2026-11-02 23:12:00.321053799 +0000 UTC m=+1113.728365631" Nov 02 23:12:01.211182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:01.211068 5672 generic.go:356] "Generic (PLEG): container finished" podID="dbd1ad31-73cb-4aeb-908e-a0ea39a628b5" containerID="33b4254f27728124cb73d44c0fad01d76b7d42696493f77ac79411e52480370c" exitCode=0 Nov 02 23:12:01.211182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:01.211142 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" event={"ID":"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5","Type":"ContainerDied","Data":"33b4254f27728124cb73d44c0fad01d76b7d42696493f77ac79411e52480370c"} Nov 02 23:12:01.212615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:01.212591 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" event={"ID":"97473a88-1d49-4873-970e-914e29a6908f","Type":"ContainerStarted","Data":"f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620"} Nov 02 23:12:01.212741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:01.212622 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" event={"ID":"97473a88-1d49-4873-970e-914e29a6908f","Type":"ContainerStarted","Data":"bdb3070af0b31ecb473259c85eba1d36021de5588870944dccd108e2f616b870"} Nov 02 23:12:01.428809 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:01.428758 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podStartSLOduration=2.4287436749999998 podStartE2EDuration="2.428743675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:01.426588549 +0000 UTC m=+1114.833900402" watchObservedRunningTime="2026-11-02 23:12:01.428743675 +0000 UTC m=+1114.836055505" Nov 02 23:12:01.873630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:01.873599 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:12:02.359343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:02.359314 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:12:02.391993 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:02.391965 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-kube-api-access-lr98w\" (UniqueName: \"kubernetes.io/projected/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-kube-api-access-lr98w\") pod \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " Nov 02 23:12:02.392121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:02.392071 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-openshift-ca\") pod \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " Nov 02 23:12:02.392121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:02.392107 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-xks-ca\") pod \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\" (UID: \"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5\") " Nov 02 23:12:02.392509 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:02.392481 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-openshift-ca" pod "dbd1ad31-73cb-4aeb-908e-a0ea39a628b5" (UID: "dbd1ad31-73cb-4aeb-908e-a0ea39a628b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:02.393968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:02.393938 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-kube-api-access-lr98w" pod "dbd1ad31-73cb-4aeb-908e-a0ea39a628b5" (UID: "dbd1ad31-73cb-4aeb-908e-a0ea39a628b5"). InnerVolumeSpecName "kube-api-access-lr98w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:02.393968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:02.393957 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-xks-ca" pod "dbd1ad31-73cb-4aeb-908e-a0ea39a628b5" (UID: "dbd1ad31-73cb-4aeb-908e-a0ea39a628b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:02.493400 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:02.493312 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr98w\" (UniqueName: \"kubernetes.io/projected/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-kube-api-access-lr98w\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:02.493400 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:02.493350 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:02.493400 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:02.493366 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:03.109923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:03.109895 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:12:03.225693 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:03.225666 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" Nov 02 23:12:03.225861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:03.225693 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd" event={"ID":"dbd1ad31-73cb-4aeb-908e-a0ea39a628b5","Type":"ContainerDied","Data":"166b9a47508f1371d44dc5e3ed97bb7819844f76f4f79f8091833394e4a0149f"} Nov 02 23:12:03.225861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:03.225739 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="166b9a47508f1371d44dc5e3ed97bb7819844f76f4f79f8091833394e4a0149f" Nov 02 23:12:08.186937 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.186901 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r"] Nov 02 23:12:08.187615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.187596 5672 cpu_manager.go:414] "containerMap: removing container" podUID="dbd1ad31-73cb-4aeb-908e-a0ea39a628b5" containerName="revoke-keys" Nov 02 23:12:08.187667 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.187619 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd1ad31-73cb-4aeb-908e-a0ea39a628b5" containerName="revoke-keys" Nov 02 23:12:08.187781 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.187761 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd1ad31-73cb-4aeb-908e-a0ea39a628b5" containerName="revoke-keys" Nov 02 23:12:08.192114 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.192094 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:08.244350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.244314 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:12:08.245563 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.245536 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r"] Nov 02 23:12:08.272136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.272088 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b989047c-5268-4996-888d-aac949c546ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9e88ea5-x259r\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:08.272313 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.272177 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b989047c-5268-4996-888d-aac949c546ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9e88ea5-x259r\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:08.272386 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.272332 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kxlq\" (UniqueName: \"kubernetes.io/projected/b989047c-5268-4996-888d-aac949c546ec-kube-api-access-6kxlq\") pod \"maas-api-revoke-sub-f9e88ea5-x259r\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:08.287729 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.287705 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:12:08.287729 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.287719 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:12:08.287881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.287789 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:12:08.373931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.373892 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kxlq\" (UniqueName: \"kubernetes.io/projected/b989047c-5268-4996-888d-aac949c546ec-kube-api-access-6kxlq\") pod \"maas-api-revoke-sub-f9e88ea5-x259r\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:08.374122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.374019 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b989047c-5268-4996-888d-aac949c546ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9e88ea5-x259r\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:08.374122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.374052 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b989047c-5268-4996-888d-aac949c546ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9e88ea5-x259r\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:08.374786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.374758 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b989047c-5268-4996-888d-aac949c546ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9e88ea5-x259r\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:08.376386 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.376361 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b989047c-5268-4996-888d-aac949c546ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9e88ea5-x259r\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:08.431552 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.431519 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kxlq\" (UniqueName: \"kubernetes.io/projected/b989047c-5268-4996-888d-aac949c546ec-kube-api-access-6kxlq\") pod \"maas-api-revoke-sub-f9e88ea5-x259r\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:08.505461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.505433 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:08.873976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:08.873943 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r"] Nov 02 23:12:08.875686 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:12:08.875657 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb989047c_5268_4996_888d_aac949c546ec.slice/crio-0a9a2772c024e8ac8938112d22e8a2c738a14a369d9c756c50a9dca9df5b34f6 WatchSource:0}: Error finding container 0a9a2772c024e8ac8938112d22e8a2c738a14a369d9c756c50a9dca9df5b34f6: Status 404 returned error can't find the container with id 0a9a2772c024e8ac8938112d22e8a2c738a14a369d9c756c50a9dca9df5b34f6 Nov 02 23:12:09.303176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.303117 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" event={"ID":"b989047c-5268-4996-888d-aac949c546ec","Type":"ContainerStarted","Data":"756da09516285c7f5711e0b08325eda4c25b4be3ff662f91a4fc92d26f4b1d91"} Nov 02 23:12:09.303624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.303190 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" event={"ID":"b989047c-5268-4996-888d-aac949c546ec","Type":"ContainerStarted","Data":"0a9a2772c024e8ac8938112d22e8a2c738a14a369d9c756c50a9dca9df5b34f6"} Nov 02 23:12:09.411556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.411469 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" podStartSLOduration=1.411455851 podStartE2EDuration="1.411455851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:09.410574371 +0000 UTC m=+1122.817886201" watchObservedRunningTime="2026-11-02 23:12:09.411455851 +0000 UTC m=+1122.818767675" Nov 02 23:12:09.527605 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.527573 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p"] Nov 02 23:12:09.531617 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.531593 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:09.584924 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.584893 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p"] Nov 02 23:12:09.587889 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.587855 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p2pd\" (UniqueName: \"kubernetes.io/projected/b9f20596-326f-4689-89cb-5587dfadf663-kube-api-access-4p2pd\") pod \"maas-api-revoke-sub-f1f0022f-kkw5p\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:09.588045 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.588023 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f0022f-kkw5p\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:09.588132 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.588115 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f0022f-kkw5p\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:09.689642 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.689607 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f0022f-kkw5p\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:09.689808 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.689748 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f0022f-kkw5p\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:09.689883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.689858 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2pd\" (UniqueName: \"kubernetes.io/projected/b9f20596-326f-4689-89cb-5587dfadf663-kube-api-access-4p2pd\") pod \"maas-api-revoke-sub-f1f0022f-kkw5p\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:09.690431 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.690403 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f0022f-kkw5p\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:09.692058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.692036 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f0022f-kkw5p\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:09.743349 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.743323 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2pd\" (UniqueName: \"kubernetes.io/projected/b9f20596-326f-4689-89cb-5587dfadf663-kube-api-access-4p2pd\") pod \"maas-api-revoke-sub-f1f0022f-kkw5p\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:09.845826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.845790 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:09.873006 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.872971 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:12:09.885706 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:09.885683 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:12:10.214117 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:10.214049 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p"] Nov 02 23:12:10.216518 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:12:10.216480 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f20596_326f_4689_89cb_5587dfadf663.slice/crio-80611bdd1b03e80af784af01f1ac8aabf57d98f15645d868cb1ee80c83670516 WatchSource:0}: Error finding container 80611bdd1b03e80af784af01f1ac8aabf57d98f15645d868cb1ee80c83670516: Status 404 returned error can't find the container with id 80611bdd1b03e80af784af01f1ac8aabf57d98f15645d868cb1ee80c83670516 Nov 02 23:12:10.310933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:10.310889 5672 generic.go:356] "Generic (PLEG): container finished" podID="b989047c-5268-4996-888d-aac949c546ec" containerID="756da09516285c7f5711e0b08325eda4c25b4be3ff662f91a4fc92d26f4b1d91" exitCode=0 Nov 02 23:12:10.311363 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:10.310977 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" event={"ID":"b989047c-5268-4996-888d-aac949c546ec","Type":"ContainerDied","Data":"756da09516285c7f5711e0b08325eda4c25b4be3ff662f91a4fc92d26f4b1d91"} Nov 02 23:12:10.312388 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:10.312365 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" event={"ID":"b9f20596-326f-4689-89cb-5587dfadf663","Type":"ContainerStarted","Data":"80611bdd1b03e80af784af01f1ac8aabf57d98f15645d868cb1ee80c83670516"} Nov 02 23:12:10.324041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:10.324022 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:12:11.321165 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.321119 5672 generic.go:356] "Generic (PLEG): container finished" podID="b9f20596-326f-4689-89cb-5587dfadf663" containerID="0fdc5a43743355daed110e04dd6b1a6e945ee5e3716fe5500c280421b038c7b8" exitCode=0 Nov 02 23:12:11.321644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.321205 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" event={"ID":"b9f20596-326f-4689-89cb-5587dfadf663","Type":"ContainerDied","Data":"0fdc5a43743355daed110e04dd6b1a6e945ee5e3716fe5500c280421b038c7b8"} Nov 02 23:12:11.462621 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.462597 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:11.611509 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.611476 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b989047c-5268-4996-888d-aac949c546ec-kube-api-access-6kxlq\" (UniqueName: \"kubernetes.io/projected/b989047c-5268-4996-888d-aac949c546ec-kube-api-access-6kxlq\") pod \"b989047c-5268-4996-888d-aac949c546ec\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " Nov 02 23:12:11.611670 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.611637 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b989047c-5268-4996-888d-aac949c546ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b989047c-5268-4996-888d-aac949c546ec-maas-api-openshift-ca\") pod \"b989047c-5268-4996-888d-aac949c546ec\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " Nov 02 23:12:11.611715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.611684 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b989047c-5268-4996-888d-aac949c546ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b989047c-5268-4996-888d-aac949c546ec-maas-api-xks-ca\") pod \"b989047c-5268-4996-888d-aac949c546ec\" (UID: \"b989047c-5268-4996-888d-aac949c546ec\") " Nov 02 23:12:11.612007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.611975 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b989047c-5268-4996-888d-aac949c546ec-maas-api-openshift-ca" pod "b989047c-5268-4996-888d-aac949c546ec" (UID: "b989047c-5268-4996-888d-aac949c546ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:11.613443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.613421 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b989047c-5268-4996-888d-aac949c546ec-kube-api-access-6kxlq" pod "b989047c-5268-4996-888d-aac949c546ec" (UID: "b989047c-5268-4996-888d-aac949c546ec"). InnerVolumeSpecName "kube-api-access-6kxlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:11.613661 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.613639 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b989047c-5268-4996-888d-aac949c546ec-maas-api-xks-ca" pod "b989047c-5268-4996-888d-aac949c546ec" (UID: "b989047c-5268-4996-888d-aac949c546ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:11.713238 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.713169 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b989047c-5268-4996-888d-aac949c546ec-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:11.713238 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.713197 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b989047c-5268-4996-888d-aac949c546ec-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:11.713238 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:11.713211 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kxlq\" (UniqueName: \"kubernetes.io/projected/b989047c-5268-4996-888d-aac949c546ec-kube-api-access-6kxlq\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:12.326616 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.326585 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr"] Nov 02 23:12:12.327132 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.327113 5672 cpu_manager.go:414] "containerMap: removing container" podUID="b989047c-5268-4996-888d-aac949c546ec" containerName="revoke-keys" Nov 02 23:12:12.327193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.327134 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b989047c-5268-4996-888d-aac949c546ec" containerName="revoke-keys" Nov 02 23:12:12.327272 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.327261 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="b989047c-5268-4996-888d-aac949c546ec" containerName="revoke-keys" Nov 02 23:12:12.330236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.330213 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" Nov 02 23:12:12.330678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.330498 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r" event={"ID":"b989047c-5268-4996-888d-aac949c546ec","Type":"ContainerDied","Data":"0a9a2772c024e8ac8938112d22e8a2c738a14a369d9c756c50a9dca9df5b34f6"} Nov 02 23:12:12.330678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.330536 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a9a2772c024e8ac8938112d22e8a2c738a14a369d9c756c50a9dca9df5b34f6" Nov 02 23:12:12.330678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.330595 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.421875 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.421841 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.421875 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.421881 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p55l2\" (UniqueName: \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-kube-api-access-p55l2\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.422118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.421904 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.422118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.422018 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.422118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.422045 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53e1f95-aa31-4384-a917-f1392d50907d-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.422323 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.422229 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.422450 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.422426 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.422515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.422472 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53e1f95-aa31-4384-a917-f1392d50907d-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.422571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.422551 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.428492 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.428467 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr"] Nov 02 23:12:12.488949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.488922 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:12.524180 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.524134 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.524330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.524267 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.524330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.524301 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53e1f95-aa31-4384-a917-f1392d50907d-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.524455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.524332 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.524455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.524379 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.524455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.524408 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p55l2\" (UniqueName: \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-kube-api-access-p55l2\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.524455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.524437 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.524649 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.524559 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.524704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.524640 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.524704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.524689 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53e1f95-aa31-4384-a917-f1392d50907d-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.524810 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.524689 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.525059 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.525039 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53e1f95-aa31-4384-a917-f1392d50907d-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.525175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.525059 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.525175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.525103 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.526945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.526924 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.527086 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.527063 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53e1f95-aa31-4384-a917-f1392d50907d-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.577990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.577915 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.578122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.578023 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p55l2\" (UniqueName: \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-kube-api-access-p55l2\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.626121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.626090 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-xks-ca\") pod \"b9f20596-326f-4689-89cb-5587dfadf663\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " Nov 02 23:12:12.626283 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.626164 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f20596-326f-4689-89cb-5587dfadf663-kube-api-access-4p2pd\" (UniqueName: \"kubernetes.io/projected/b9f20596-326f-4689-89cb-5587dfadf663-kube-api-access-4p2pd\") pod \"b9f20596-326f-4689-89cb-5587dfadf663\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " Nov 02 23:12:12.626345 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.626333 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-openshift-ca\") pod \"b9f20596-326f-4689-89cb-5587dfadf663\" (UID: \"b9f20596-326f-4689-89cb-5587dfadf663\") " Nov 02 23:12:12.626784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.626744 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-openshift-ca" pod "b9f20596-326f-4689-89cb-5587dfadf663" (UID: "b9f20596-326f-4689-89cb-5587dfadf663"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:12.628339 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.628313 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-xks-ca" pod "b9f20596-326f-4689-89cb-5587dfadf663" (UID: "b9f20596-326f-4689-89cb-5587dfadf663"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:12.628640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.628617 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f20596-326f-4689-89cb-5587dfadf663-kube-api-access-4p2pd" pod "b9f20596-326f-4689-89cb-5587dfadf663" (UID: "b9f20596-326f-4689-89cb-5587dfadf663"). InnerVolumeSpecName "kube-api-access-4p2pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:12.647065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.647034 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:12.728378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.728349 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:12.728378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.728373 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p2pd\" (UniqueName: \"kubernetes.io/projected/b9f20596-326f-4689-89cb-5587dfadf663-kube-api-access-4p2pd\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:12.728378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:12.728383 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f20596-326f-4689-89cb-5587dfadf663-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:13.061317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:13.061281 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr"] Nov 02 23:12:13.063377 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:12:13.063351 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53e1f95_aa31_4384_a917_f1392d50907d.slice/crio-38b321bf6c0b95a766362348aed03ab2be6337a2874deb298a2c2615af745d42 WatchSource:0}: Error finding container 38b321bf6c0b95a766362348aed03ab2be6337a2874deb298a2c2615af745d42: Status 404 returned error can't find the container with id 38b321bf6c0b95a766362348aed03ab2be6337a2874deb298a2c2615af745d42 Nov 02 23:12:13.065989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:13.065957 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:13.066077 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:13.066018 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:13.066077 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:13.066047 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:13.340222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:13.339297 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" Nov 02 23:12:13.340222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:13.339298 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p" event={"ID":"b9f20596-326f-4689-89cb-5587dfadf663","Type":"ContainerDied","Data":"80611bdd1b03e80af784af01f1ac8aabf57d98f15645d868cb1ee80c83670516"} Nov 02 23:12:13.340222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:13.339433 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80611bdd1b03e80af784af01f1ac8aabf57d98f15645d868cb1ee80c83670516" Nov 02 23:12:13.349187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:13.341835 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" event={"ID":"b53e1f95-aa31-4384-a917-f1392d50907d","Type":"ContainerStarted","Data":"fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a"} Nov 02 23:12:13.349187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:13.341886 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" event={"ID":"b53e1f95-aa31-4384-a917-f1392d50907d","Type":"ContainerStarted","Data":"38b321bf6c0b95a766362348aed03ab2be6337a2874deb298a2c2615af745d42"} Nov 02 23:12:13.453185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:13.453114 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" podStartSLOduration=1.453096577 podStartE2EDuration="1.453096577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:13.451661086 +0000 UTC m=+1126.858972952" watchObservedRunningTime="2026-11-02 23:12:13.453096577 +0000 UTC m=+1126.860408407" Nov 02 23:12:13.651226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:13.650257 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:14.651004 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:14.650971 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:12:14.651424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:14.651024 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 23:12:15.649543 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:15.649503 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:12:15.649767 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:15.649574 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 23:12:16.648538 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:16.648487 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:12:16.649090 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:16.649055 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 23:12:17.652799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:17.652764 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:12:17.653399 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:17.653354 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 23:12:18.650068 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:18.649473 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:12:18.650068 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:18.649531 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 23:12:19.043002 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:19.042966 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp"] Nov 02 23:12:19.089372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:19.089338 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3acc52c0-gmqnp"] Nov 02 23:12:19.647967 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:19.647930 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 23:12:19.648117 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:19.647992 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 23:12:20.043438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:20.043402 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k"] Nov 02 23:12:20.046195 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:20.046169 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8781ed61-qfv9k"] Nov 02 23:12:20.652188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:20.652137 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:20.652595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:20.652569 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:20.653569 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:20.653549 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:12:20.713037 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:20.713006 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d965abc-e8a2-4476-b130-9091c8197a15" path="/var/lib/kubelet/pods/1d965abc-e8a2-4476-b130-9091c8197a15/volumes" Nov 02 23:12:20.713611 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:20.713593 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f387b94a-653b-4756-ba6a-c599406b1abf" path="/var/lib/kubelet/pods/f387b94a-653b-4756-ba6a-c599406b1abf/volumes" Nov 02 23:12:25.739122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.739075 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r"] Nov 02 23:12:25.739831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.739806 5672 cpu_manager.go:414] "containerMap: removing container" podUID="b9f20596-326f-4689-89cb-5587dfadf663" containerName="revoke-keys" Nov 02 23:12:25.739916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.739835 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f20596-326f-4689-89cb-5587dfadf663" containerName="revoke-keys" Nov 02 23:12:25.740014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.740000 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f20596-326f-4689-89cb-5587dfadf663" containerName="revoke-keys" Nov 02 23:12:25.745629 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.745597 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:25.797854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.796842 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r"] Nov 02 23:12:25.801171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.798762 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-454wq\" (UniqueName: \"kubernetes.io/projected/805b59b6-af3e-48bc-97a7-3a62db23ac6e-kube-api-access-454wq\") pod \"maas-api-revoke-sub-9f253b3a-mrr9r\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:25.801171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.798898 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f253b3a-mrr9r\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:25.801171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.799004 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f253b3a-mrr9r\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:25.843928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.843889 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:12:25.844285 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.844264 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:12:25.844546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.844527 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:12:25.844761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.844743 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:12:25.900357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.900327 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f253b3a-mrr9r\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:25.900624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.900608 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-454wq\" (UniqueName: \"kubernetes.io/projected/805b59b6-af3e-48bc-97a7-3a62db23ac6e-kube-api-access-454wq\") pod \"maas-api-revoke-sub-9f253b3a-mrr9r\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:25.900798 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.900787 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f253b3a-mrr9r\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:25.901932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.901910 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f253b3a-mrr9r\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:25.906074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:25.906054 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f253b3a-mrr9r\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:26.007302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.007215 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-454wq\" (UniqueName: \"kubernetes.io/projected/805b59b6-af3e-48bc-97a7-3a62db23ac6e-kube-api-access-454wq\") pod \"maas-api-revoke-sub-9f253b3a-mrr9r\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:26.068377 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.068341 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:26.299521 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:12:26.299400 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:12:26.912222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.910677 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk"] Nov 02 23:12:26.950173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.926114 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:26.983172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.977174 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:26.983172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.977254 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:26.983172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.977321 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:26.983172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.977369 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvvmx\" (UniqueName: \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-kube-api-access-bvvmx\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:26.983172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.977474 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:26.983172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.977510 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:26.983172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.977587 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:26.983172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.977620 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:26.983172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:26.977658 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.052242 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.052209 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk"] Nov 02 23:12:27.095172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.078807 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.095172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.078886 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.095172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.078927 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvvmx\" (UniqueName: \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-kube-api-access-bvvmx\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.095172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.079033 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.095172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.079064 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.095172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.079166 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.095172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.079196 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.095172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.079224 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.095172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.079310 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.095172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.079756 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.133171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.132507 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.133171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.132796 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.134521 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.133865 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.134521 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.134130 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.217219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.211364 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.265449 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.225680 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.330703 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.329357 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r"] Nov 02 23:12:27.400173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.400019 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvvmx\" (UniqueName: \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-kube-api-access-bvvmx\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.536173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.532481 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.621174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.620925 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" event={"ID":"805b59b6-af3e-48bc-97a7-3a62db23ac6e","Type":"ContainerStarted","Data":"1fefe7b03c508761900e98313e48c9f822fd413348cf229474d4b3fd65552630"} Nov 02 23:12:27.645172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.635662 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:27.707232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:27.684814 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:28.277543 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.277447 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 23:12:28.422889 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.382521 5672 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:28.422889 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.403651 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:28.510275 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.510232 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk"] Nov 02 23:12:28.552817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.534053 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:28.552817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.550872 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:28.552817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.550953 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:28.552817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.551006 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:28.643178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.640594 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" event={"ID":"034fd4f3-3ede-4e63-a1c7-61bc593869ac","Type":"ContainerStarted","Data":"c446b460219ac3351cf709879053e1f66a796a9d1cbdb2190354b985c6fa0ee7"} Nov 02 23:12:28.643178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.642781 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" event={"ID":"805b59b6-af3e-48bc-97a7-3a62db23ac6e","Type":"ContainerStarted","Data":"ebf53c9032f168bf1b9ede4199c5f95e3963cfe3833dc5d3c1793296639dda7c"} Nov 02 23:12:28.721822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.721790 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg"] Nov 02 23:12:28.727383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.727358 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.753069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.753041 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5"] Nov 02 23:12:28.758968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.758940 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.775419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.767981 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" podStartSLOduration=3.767964162 podStartE2EDuration="3.767964162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:28.766726491 +0000 UTC m=+1142.174038322" watchObservedRunningTime="2026-11-02 23:12:28.767964162 +0000 UTC m=+1142.175275993" Nov 02 23:12:28.776098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.776073 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg"] Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820002 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820044 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820100 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820131 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820183 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820221 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820256 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820287 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c30e52-e171-45c7-948b-f72af978cd59-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820760 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820803 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820820 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820877 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jn8t\" (UniqueName: \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-kube-api-access-9jn8t\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820906 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820925 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zfd\" (UniqueName: \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-kube-api-access-29zfd\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.820958 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.821507 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.821547 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c30e52-e171-45c7-948b-f72af978cd59-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.821799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.821618 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.887172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.883588 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5"] Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.922734 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jn8t\" (UniqueName: \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-kube-api-access-9jn8t\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.922779 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.922796 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zfd\" (UniqueName: \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-kube-api-access-29zfd\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.922820 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.922837 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.922858 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c30e52-e171-45c7-948b-f72af978cd59-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.922876 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.922916 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.922931 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.922962 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.922981 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.923002 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.923034 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.923056 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.923074 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c30e52-e171-45c7-948b-f72af978cd59-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.923122 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.923195 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.923222 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.923793 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.923898 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.924043 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.927434 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.927697 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.928188 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.930749 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.931016 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.931014 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c30e52-e171-45c7-948b-f72af978cd59-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.935598 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c30e52-e171-45c7-948b-f72af978cd59-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.936926 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.948768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.939978 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.956475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.950142 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.956848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.956779 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.979241 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.976188 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2"] Nov 02 23:12:28.980358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.980321 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:28.981019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.981000 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:28.984179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.984041 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.985404 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.985381 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jn8t\" (UniqueName: \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-kube-api-access-9jn8t\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:28.986165 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:28.986104 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zfd\" (UniqueName: \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-kube-api-access-29zfd\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:29.026410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.024301 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw45k\" (UniqueName: \"kubernetes.io/projected/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-kube-api-access-lw45k\") pod \"maas-api-revoke-sub-d16e4029-l5sr2\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:29.026410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.024401 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16e4029-l5sr2\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:29.026410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.024503 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16e4029-l5sr2\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:29.042765 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.039105 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2"] Nov 02 23:12:29.070694 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.070610 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:29.095059 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.094974 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:29.125582 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.125536 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16e4029-l5sr2\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:29.125749 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.125638 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw45k\" (UniqueName: \"kubernetes.io/projected/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-kube-api-access-lw45k\") pod \"maas-api-revoke-sub-d16e4029-l5sr2\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:29.125749 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.125689 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16e4029-l5sr2\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:29.126971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.126923 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16e4029-l5sr2\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:29.130071 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.130049 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16e4029-l5sr2\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:29.183053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.182969 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw45k\" (UniqueName: \"kubernetes.io/projected/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-kube-api-access-lw45k\") pod \"maas-api-revoke-sub-d16e4029-l5sr2\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:29.343789 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.343707 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:29.468068 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.468034 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg"] Nov 02 23:12:29.470887 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:12:29.470855 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdcf2cb_a12c_4a89_b1e2_9774768069e7.slice/crio-b617184bc7e1ea7251b1b2965ecab61b09ad8bd6823b8208aeb2d0003c9ada0c WatchSource:0}: Error finding container b617184bc7e1ea7251b1b2965ecab61b09ad8bd6823b8208aeb2d0003c9ada0c: Status 404 returned error can't find the container with id b617184bc7e1ea7251b1b2965ecab61b09ad8bd6823b8208aeb2d0003c9ada0c Nov 02 23:12:29.474851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.474039 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:29.474851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.474123 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:29.474851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.474213 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:29.513646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.513438 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5"] Nov 02 23:12:29.523344 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:12:29.523311 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c30e52_e171_45c7_948b_f72af978cd59.slice/crio-146de0c0366d20883bdf0d7b7cdf060add392ec72f7d23dac46d0fee496f0074 WatchSource:0}: Error finding container 146de0c0366d20883bdf0d7b7cdf060add392ec72f7d23dac46d0fee496f0074: Status 404 returned error can't find the container with id 146de0c0366d20883bdf0d7b7cdf060add392ec72f7d23dac46d0fee496f0074 Nov 02 23:12:29.526366 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.526327 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:29.526448 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.526419 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:29.526499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.526464 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:29.684578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.676544 5672 generic.go:356] "Generic (PLEG): container finished" podID="805b59b6-af3e-48bc-97a7-3a62db23ac6e" containerID="ebf53c9032f168bf1b9ede4199c5f95e3963cfe3833dc5d3c1793296639dda7c" exitCode=0 Nov 02 23:12:29.684578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.676681 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" event={"ID":"805b59b6-af3e-48bc-97a7-3a62db23ac6e","Type":"ContainerDied","Data":"ebf53c9032f168bf1b9ede4199c5f95e3963cfe3833dc5d3c1793296639dda7c"} Nov 02 23:12:29.684578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.679560 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" event={"ID":"7fdcf2cb-a12c-4a89-b1e2-9774768069e7","Type":"ContainerStarted","Data":"b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf"} Nov 02 23:12:29.684578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.679611 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" event={"ID":"7fdcf2cb-a12c-4a89-b1e2-9774768069e7","Type":"ContainerStarted","Data":"b617184bc7e1ea7251b1b2965ecab61b09ad8bd6823b8208aeb2d0003c9ada0c"} Nov 02 23:12:29.684578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.682827 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" event={"ID":"62c30e52-e171-45c7-948b-f72af978cd59","Type":"ContainerStarted","Data":"ae391b5fb0a5f2a6bd3e1fcff6970965cf3c53cb35355db7bae92e1002cc2b05"} Nov 02 23:12:29.684578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.682882 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" event={"ID":"62c30e52-e171-45c7-948b-f72af978cd59","Type":"ContainerStarted","Data":"146de0c0366d20883bdf0d7b7cdf060add392ec72f7d23dac46d0fee496f0074"} Nov 02 23:12:29.685514 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.685490 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" event={"ID":"034fd4f3-3ede-4e63-a1c7-61bc593869ac","Type":"ContainerStarted","Data":"2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a"} Nov 02 23:12:29.726674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.726645 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2"] Nov 02 23:12:29.902886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:29.902783 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" podStartSLOduration=3.90276919 podStartE2EDuration="3.90276919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:29.899574936 +0000 UTC m=+1143.306886766" watchObservedRunningTime="2026-11-02 23:12:29.90276919 +0000 UTC m=+1143.310081090" Nov 02 23:12:30.009652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.009566 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podStartSLOduration=2.009548633 podStartE2EDuration="2.009548633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:30.00754007 +0000 UTC m=+1143.414851902" watchObservedRunningTime="2026-11-02 23:12:30.009548633 +0000 UTC m=+1143.416860463" Nov 02 23:12:30.071552 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.071518 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:30.073375 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.073341 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:30.073555 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.073398 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:12:30.095413 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.095377 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:30.096803 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.096776 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:30.096916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.096826 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:12:30.119307 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.119262 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" podStartSLOduration=2.119247346 podStartE2EDuration="2.119247346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:30.116811872 +0000 UTC m=+1143.524123702" watchObservedRunningTime="2026-11-02 23:12:30.119247346 +0000 UTC m=+1143.526559175" Nov 02 23:12:30.637522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.637486 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:30.641306 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.641272 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:12:30.641585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.641556 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" podUID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 23:12:30.829350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.829317 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" event={"ID":"b16804ae-5327-46aa-82d3-d3df5ab9c2c7","Type":"ContainerStarted","Data":"52448a9625eca0e9bad8c3a9c80b99ed2ceccb4c94afda4f6612c0ade20b2379"} Nov 02 23:12:30.829537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:30.829520 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" event={"ID":"b16804ae-5327-46aa-82d3-d3df5ab9c2c7","Type":"ContainerStarted","Data":"e4549c4ee3906b178551ae679870e1dc349105666e13ebe3b8a3c561f8bb9910"} Nov 02 23:12:31.073174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.072636 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:31.073174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.072710 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:12:31.125098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.105775 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:31.125098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.105854 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:12:31.688646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.688611 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:12:31.798172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.747372 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" podUID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 23:12:31.878485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.878452 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:31.923587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.923497 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" event={"ID":"805b59b6-af3e-48bc-97a7-3a62db23ac6e","Type":"ContainerDied","Data":"1fefe7b03c508761900e98313e48c9f822fd413348cf229474d4b3fd65552630"} Nov 02 23:12:31.923843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.923824 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fefe7b03c508761900e98313e48c9f822fd413348cf229474d4b3fd65552630" Nov 02 23:12:31.924029 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.924016 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r" Nov 02 23:12:31.949030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.948998 5672 generic.go:356] "Generic (PLEG): container finished" podID="b16804ae-5327-46aa-82d3-d3df5ab9c2c7" containerID="52448a9625eca0e9bad8c3a9c80b99ed2ceccb4c94afda4f6612c0ade20b2379" exitCode=0 Nov 02 23:12:31.950256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.949336 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" event={"ID":"b16804ae-5327-46aa-82d3-d3df5ab9c2c7","Type":"ContainerDied","Data":"52448a9625eca0e9bad8c3a9c80b99ed2ceccb4c94afda4f6612c0ade20b2379"} Nov 02 23:12:31.992951 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:31.992894 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" podStartSLOduration=3.992875175 podStartE2EDuration="3.992875175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:30.980363971 +0000 UTC m=+1144.387675805" watchObservedRunningTime="2026-11-02 23:12:31.992875175 +0000 UTC m=+1145.400187007" Nov 02 23:12:32.009763 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.009730 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805b59b6-af3e-48bc-97a7-3a62db23ac6e-kube-api-access-454wq\" (UniqueName: \"kubernetes.io/projected/805b59b6-af3e-48bc-97a7-3a62db23ac6e-kube-api-access-454wq\") pod \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " Nov 02 23:12:32.010502 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.010479 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-openshift-ca\") pod \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " Nov 02 23:12:32.011136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.011022 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-openshift-ca" pod "805b59b6-af3e-48bc-97a7-3a62db23ac6e" (UID: "805b59b6-af3e-48bc-97a7-3a62db23ac6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:32.011355 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.011336 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-xks-ca\") pod \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\" (UID: \"805b59b6-af3e-48bc-97a7-3a62db23ac6e\") " Nov 02 23:12:32.013526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.013506 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:32.041178 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:12:32.015996 5672 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16804ae_5327_46aa_82d3_d3df5ab9c2c7.slice/crio-52448a9625eca0e9bad8c3a9c80b99ed2ceccb4c94afda4f6612c0ade20b2379.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16804ae_5327_46aa_82d3_d3df5ab9c2c7.slice/crio-conmon-52448a9625eca0e9bad8c3a9c80b99ed2ceccb4c94afda4f6612c0ade20b2379.scope\": RecentStats: unable to find data in memory cache]" Nov 02 23:12:32.086401 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.086358 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805b59b6-af3e-48bc-97a7-3a62db23ac6e-kube-api-access-454wq" pod "805b59b6-af3e-48bc-97a7-3a62db23ac6e" (UID: "805b59b6-af3e-48bc-97a7-3a62db23ac6e"). InnerVolumeSpecName "kube-api-access-454wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:32.111463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.089423 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-xks-ca" pod "805b59b6-af3e-48bc-97a7-3a62db23ac6e" (UID: "805b59b6-af3e-48bc-97a7-3a62db23ac6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:32.111463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.090536 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:32.111463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.090595 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:12:32.151289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.116025 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805b59b6-af3e-48bc-97a7-3a62db23ac6e-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:32.151289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.116064 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-454wq\" (UniqueName: \"kubernetes.io/projected/805b59b6-af3e-48bc-97a7-3a62db23ac6e-kube-api-access-454wq\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:32.151289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.116395 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:32.151289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.116442 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:12:32.649644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.649601 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:12:32.649920 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:32.649887 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" podUID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 23:12:33.086197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.086141 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:33.086832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.086801 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:12:33.099654 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.099537 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:33.099654 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.099587 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:12:33.677385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.677292 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:12:33.677643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.677603 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" podUID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 23:12:33.814012 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.813983 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:33.919639 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.919605 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-openshift-ca\") pod \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " Nov 02 23:12:33.919808 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.919684 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-kube-api-access-lw45k\" (UniqueName: \"kubernetes.io/projected/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-kube-api-access-lw45k\") pod \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " Nov 02 23:12:33.919850 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.919821 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-xks-ca\") pod \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\" (UID: \"b16804ae-5327-46aa-82d3-d3df5ab9c2c7\") " Nov 02 23:12:33.920193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.920168 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-openshift-ca" pod "b16804ae-5327-46aa-82d3-d3df5ab9c2c7" (UID: "b16804ae-5327-46aa-82d3-d3df5ab9c2c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:33.921858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.921833 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-xks-ca" pod "b16804ae-5327-46aa-82d3-d3df5ab9c2c7" (UID: "b16804ae-5327-46aa-82d3-d3df5ab9c2c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:33.922270 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:33.922249 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-kube-api-access-lw45k" pod "b16804ae-5327-46aa-82d3-d3df5ab9c2c7" (UID: "b16804ae-5327-46aa-82d3-d3df5ab9c2c7"). InnerVolumeSpecName "kube-api-access-lw45k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:34.022439 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.022391 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:34.022703 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.022668 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw45k\" (UniqueName: \"kubernetes.io/projected/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-kube-api-access-lw45k\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:34.022820 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.022804 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b16804ae-5327-46aa-82d3-d3df5ab9c2c7-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:34.099467 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.099430 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:34.100001 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.099971 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:12:34.100742 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.100719 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:34.100892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.100872 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:12:34.149906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.149868 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" event={"ID":"b16804ae-5327-46aa-82d3-d3df5ab9c2c7","Type":"ContainerDied","Data":"e4549c4ee3906b178551ae679870e1dc349105666e13ebe3b8a3c561f8bb9910"} Nov 02 23:12:34.150329 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.150308 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4549c4ee3906b178551ae679870e1dc349105666e13ebe3b8a3c561f8bb9910" Nov 02 23:12:34.150519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.150486 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2" Nov 02 23:12:34.213926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.213883 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2"] Nov 02 23:12:34.268173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.248477 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6810ea-22sx2"] Nov 02 23:12:34.638554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.638518 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:12:34.638803 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.638776 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" podUID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 23:12:34.729173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:34.724251 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1929b2-4756-4bbe-986c-12d83d83d3c2" path="/var/lib/kubelet/pods/9c1929b2-4756-4bbe-986c-12d83d83d3c2/volumes" Nov 02 23:12:35.077344 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.077293 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:35.077674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.077622 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:12:35.097913 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.097882 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:35.098349 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.098128 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:12:35.408546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.408459 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g"] Nov 02 23:12:35.409197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.409135 5672 cpu_manager.go:414] "containerMap: removing container" podUID="b16804ae-5327-46aa-82d3-d3df5ab9c2c7" containerName="revoke-keys" Nov 02 23:12:35.409197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.409174 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16804ae-5327-46aa-82d3-d3df5ab9c2c7" containerName="revoke-keys" Nov 02 23:12:35.409357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.409227 5672 cpu_manager.go:414] "containerMap: removing container" podUID="805b59b6-af3e-48bc-97a7-3a62db23ac6e" containerName="revoke-keys" Nov 02 23:12:35.409357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.409235 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805b59b6-af3e-48bc-97a7-3a62db23ac6e" containerName="revoke-keys" Nov 02 23:12:35.409453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.409370 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="805b59b6-af3e-48bc-97a7-3a62db23ac6e" containerName="revoke-keys" Nov 02 23:12:35.409453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.409389 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16804ae-5327-46aa-82d3-d3df5ab9c2c7" containerName="revoke-keys" Nov 02 23:12:35.414007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.413986 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:35.467174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.466563 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:12:35.467360 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.467295 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:12:35.467360 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.467315 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:12:35.481170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.469129 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g"] Nov 02 23:12:35.508207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.508165 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t6tj\" (UniqueName: \"kubernetes.io/projected/8b9755d8-1445-4e6c-bd03-d69af2280d17-kube-api-access-2t6tj\") pod \"maas-api-revoke-sub-b5482e69-p2g6g\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:35.508379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.508297 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5482e69-p2g6g\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:35.508379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.508364 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5482e69-p2g6g\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:35.510111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.509479 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:12:35.609045 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.609009 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t6tj\" (UniqueName: \"kubernetes.io/projected/8b9755d8-1445-4e6c-bd03-d69af2280d17-kube-api-access-2t6tj\") pod \"maas-api-revoke-sub-b5482e69-p2g6g\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:35.609267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.609117 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5482e69-p2g6g\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:35.609267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.609177 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5482e69-p2g6g\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:35.609847 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.609817 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5482e69-p2g6g\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:35.613577 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.613555 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5482e69-p2g6g\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:35.637227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.637200 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 23:12:35.637487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.637450 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" podUID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 23:12:35.681020 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.680945 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t6tj\" (UniqueName: \"kubernetes.io/projected/8b9755d8-1445-4e6c-bd03-d69af2280d17-kube-api-access-2t6tj\") pod \"maas-api-revoke-sub-b5482e69-p2g6g\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:35.742546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:35.742516 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:36.072083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.072048 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:36.072460 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.072428 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:12:36.095881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.095842 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:36.096168 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.096125 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:12:36.217314 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.217273 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g"] Nov 02 23:12:36.487022 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.486987 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp"] Nov 02 23:12:36.493164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.493120 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:36.521272 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.521215 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fcf7ecf-w6qtp\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:36.521417 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.521324 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fcf7ecf-w6qtp\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:36.521502 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.521483 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndp4h\" (UniqueName: \"kubernetes.io/projected/77d7bab5-706c-474b-a725-77fb942984ad-kube-api-access-ndp4h\") pod \"maas-api-revoke-sub-7fcf7ecf-w6qtp\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:36.564189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.563521 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp"] Nov 02 23:12:36.625866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.622849 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndp4h\" (UniqueName: \"kubernetes.io/projected/77d7bab5-706c-474b-a725-77fb942984ad-kube-api-access-ndp4h\") pod \"maas-api-revoke-sub-7fcf7ecf-w6qtp\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:36.625866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.622980 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fcf7ecf-w6qtp\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:36.625866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.623045 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fcf7ecf-w6qtp\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:36.627650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.627599 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fcf7ecf-w6qtp\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:36.630116 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.630090 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fcf7ecf-w6qtp\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:36.682278 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.682247 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndp4h\" (UniqueName: \"kubernetes.io/projected/77d7bab5-706c-474b-a725-77fb942984ad-kube-api-access-ndp4h\") pod \"maas-api-revoke-sub-7fcf7ecf-w6qtp\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:36.812847 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.812794 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:36.930771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.930739 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:36.932477 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.932453 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:36.936838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:36.936814 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:12:37.071367 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:37.071283 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 23:12:37.071367 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:37.071333 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 23:12:37.096373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:37.096340 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 23:12:37.096513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:37.096393 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 23:12:37.217701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:37.216663 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp"] Nov 02 23:12:37.234460 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:37.227721 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" event={"ID":"8b9755d8-1445-4e6c-bd03-d69af2280d17","Type":"ContainerStarted","Data":"ce0ad43b3af7bd32f98b23e1b3d7441e60af934d41155a3e9a3f76dcf3fb8298"} Nov 02 23:12:37.234460 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:37.227778 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" event={"ID":"8b9755d8-1445-4e6c-bd03-d69af2280d17","Type":"ContainerStarted","Data":"5b400b4dd5e7ffeabe9d15ec00386cbbdd7565fbf2f6350a302b82ace38b57e1"} Nov 02 23:12:37.343929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:37.343808 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" podStartSLOduration=2.34378799 podStartE2EDuration="2.34378799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:37.332965017 +0000 UTC m=+1150.740276849" watchObservedRunningTime="2026-11-02 23:12:37.34378799 +0000 UTC m=+1150.751099821" Nov 02 23:12:38.075806 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:38.075778 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:38.076242 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:38.076182 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:38.076792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:38.076775 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:12:38.100216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:38.100176 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:38.100627 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:38.100601 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:38.101301 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:38.101280 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:12:38.234369 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:38.234320 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" event={"ID":"77d7bab5-706c-474b-a725-77fb942984ad","Type":"ContainerStarted","Data":"8fdf98066a2bb3bade155269676767362862161c5189aa49f7b0756d7e551e64"} Nov 02 23:12:38.234527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:38.234390 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" event={"ID":"77d7bab5-706c-474b-a725-77fb942984ad","Type":"ContainerStarted","Data":"6a707ee5a860fdfd1bdf19695c4d5f103d5465f211b69c83abbe80b7881181fd"} Nov 02 23:12:38.236107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:38.236080 5672 generic.go:356] "Generic (PLEG): container finished" podID="8b9755d8-1445-4e6c-bd03-d69af2280d17" containerID="ce0ad43b3af7bd32f98b23e1b3d7441e60af934d41155a3e9a3f76dcf3fb8298" exitCode=0 Nov 02 23:12:38.236243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:38.236140 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" event={"ID":"8b9755d8-1445-4e6c-bd03-d69af2280d17","Type":"ContainerDied","Data":"ce0ad43b3af7bd32f98b23e1b3d7441e60af934d41155a3e9a3f76dcf3fb8298"} Nov 02 23:12:38.618304 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:38.618244 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" podStartSLOduration=2.618229645 podStartE2EDuration="2.618229645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:38.617057962 +0000 UTC m=+1152.024369797" watchObservedRunningTime="2026-11-02 23:12:38.618229645 +0000 UTC m=+1152.025541475" Nov 02 23:12:39.244628 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.244595 5672 generic.go:356] "Generic (PLEG): container finished" podID="77d7bab5-706c-474b-a725-77fb942984ad" containerID="8fdf98066a2bb3bade155269676767362862161c5189aa49f7b0756d7e551e64" exitCode=0 Nov 02 23:12:39.245104 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.244694 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" event={"ID":"77d7bab5-706c-474b-a725-77fb942984ad","Type":"ContainerDied","Data":"8fdf98066a2bb3bade155269676767362862161c5189aa49f7b0756d7e551e64"} Nov 02 23:12:39.394648 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.394623 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:39.469363 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.469326 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9755d8-1445-4e6c-bd03-d69af2280d17-kube-api-access-2t6tj\" (UniqueName: \"kubernetes.io/projected/8b9755d8-1445-4e6c-bd03-d69af2280d17-kube-api-access-2t6tj\") pod \"8b9755d8-1445-4e6c-bd03-d69af2280d17\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " Nov 02 23:12:39.469584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.469394 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-openshift-ca\") pod \"8b9755d8-1445-4e6c-bd03-d69af2280d17\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " Nov 02 23:12:39.469584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.469558 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-xks-ca\") pod \"8b9755d8-1445-4e6c-bd03-d69af2280d17\" (UID: \"8b9755d8-1445-4e6c-bd03-d69af2280d17\") " Nov 02 23:12:39.469744 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.469717 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-openshift-ca" pod "8b9755d8-1445-4e6c-bd03-d69af2280d17" (UID: "8b9755d8-1445-4e6c-bd03-d69af2280d17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:39.470241 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.470219 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:39.471592 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.471565 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9755d8-1445-4e6c-bd03-d69af2280d17-kube-api-access-2t6tj" pod "8b9755d8-1445-4e6c-bd03-d69af2280d17" (UID: "8b9755d8-1445-4e6c-bd03-d69af2280d17"). InnerVolumeSpecName "kube-api-access-2t6tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:39.471681 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.471614 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-xks-ca" pod "8b9755d8-1445-4e6c-bd03-d69af2280d17" (UID: "8b9755d8-1445-4e6c-bd03-d69af2280d17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:39.571544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.571475 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t6tj\" (UniqueName: \"kubernetes.io/projected/8b9755d8-1445-4e6c-bd03-d69af2280d17-kube-api-access-2t6tj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:39.571544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:39.571504 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b9755d8-1445-4e6c-bd03-d69af2280d17-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:40.253286 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.253257 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" Nov 02 23:12:40.253698 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.253294 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g" event={"ID":"8b9755d8-1445-4e6c-bd03-d69af2280d17","Type":"ContainerDied","Data":"5b400b4dd5e7ffeabe9d15ec00386cbbdd7565fbf2f6350a302b82ace38b57e1"} Nov 02 23:12:40.253698 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.253341 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b400b4dd5e7ffeabe9d15ec00386cbbdd7565fbf2f6350a302b82ace38b57e1" Nov 02 23:12:40.401622 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.401601 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:40.483261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.483221 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-openshift-ca\") pod \"77d7bab5-706c-474b-a725-77fb942984ad\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " Nov 02 23:12:40.483431 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.483407 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-xks-ca\") pod \"77d7bab5-706c-474b-a725-77fb942984ad\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " Nov 02 23:12:40.483494 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.483470 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d7bab5-706c-474b-a725-77fb942984ad-kube-api-access-ndp4h\" (UniqueName: \"kubernetes.io/projected/77d7bab5-706c-474b-a725-77fb942984ad-kube-api-access-ndp4h\") pod \"77d7bab5-706c-474b-a725-77fb942984ad\" (UID: \"77d7bab5-706c-474b-a725-77fb942984ad\") " Nov 02 23:12:40.483622 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.483594 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-openshift-ca" pod "77d7bab5-706c-474b-a725-77fb942984ad" (UID: "77d7bab5-706c-474b-a725-77fb942984ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:40.483897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.483881 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:40.485557 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.485529 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-xks-ca" pod "77d7bab5-706c-474b-a725-77fb942984ad" (UID: "77d7bab5-706c-474b-a725-77fb942984ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:40.485798 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.485770 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d7bab5-706c-474b-a725-77fb942984ad-kube-api-access-ndp4h" pod "77d7bab5-706c-474b-a725-77fb942984ad" (UID: "77d7bab5-706c-474b-a725-77fb942984ad"). InnerVolumeSpecName "kube-api-access-ndp4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:40.585293 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.585215 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77d7bab5-706c-474b-a725-77fb942984ad-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:40.585293 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:40.585241 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndp4h\" (UniqueName: \"kubernetes.io/projected/77d7bab5-706c-474b-a725-77fb942984ad-kube-api-access-ndp4h\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:41.064830 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:41.064794 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5"] Nov 02 23:12:41.109142 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:41.109107 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad3b5b2-zdcq5"] Nov 02 23:12:41.261597 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:41.261569 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" Nov 02 23:12:41.261597 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:41.261590 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp" event={"ID":"77d7bab5-706c-474b-a725-77fb942984ad","Type":"ContainerDied","Data":"6a707ee5a860fdfd1bdf19695c4d5f103d5465f211b69c83abbe80b7881181fd"} Nov 02 23:12:41.262035 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:41.261636 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a707ee5a860fdfd1bdf19695c4d5f103d5465f211b69c83abbe80b7881181fd" Nov 02 23:12:42.713297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:42.713255 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dfa24b3-ffd4-4893-a002-f05155afb551" path="/var/lib/kubelet/pods/5dfa24b3-ffd4-4893-a002-f05155afb551/volumes" Nov 02 23:12:45.858976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.858939 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb"] Nov 02 23:12:45.859712 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.859688 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8b9755d8-1445-4e6c-bd03-d69af2280d17" containerName="revoke-keys" Nov 02 23:12:45.859712 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.859708 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9755d8-1445-4e6c-bd03-d69af2280d17" containerName="revoke-keys" Nov 02 23:12:45.859894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.859727 5672 cpu_manager.go:414] "containerMap: removing container" podUID="77d7bab5-706c-474b-a725-77fb942984ad" containerName="revoke-keys" Nov 02 23:12:45.859894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.859733 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d7bab5-706c-474b-a725-77fb942984ad" containerName="revoke-keys" Nov 02 23:12:45.859894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.859846 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9755d8-1445-4e6c-bd03-d69af2280d17" containerName="revoke-keys" Nov 02 23:12:45.859894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.859859 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d7bab5-706c-474b-a725-77fb942984ad" containerName="revoke-keys" Nov 02 23:12:45.863389 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.863372 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:45.916636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.916581 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb"] Nov 02 23:12:45.949181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.949131 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4fhp\" (UniqueName: \"kubernetes.io/projected/9b71bea1-31a1-4e7b-b913-8f305a3af254-kube-api-access-z4fhp\") pod \"maas-api-revoke-sub-75cdb93f-rmrrb\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:45.949351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.949326 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75cdb93f-rmrrb\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:45.949530 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.949508 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75cdb93f-rmrrb\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:45.958325 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.958300 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:12:45.958325 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.958318 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:12:45.960223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.960203 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:12:45.960357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:45.960239 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:12:46.051382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.051352 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75cdb93f-rmrrb\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:46.051568 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.051484 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75cdb93f-rmrrb\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:46.051568 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.051519 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4fhp\" (UniqueName: \"kubernetes.io/projected/9b71bea1-31a1-4e7b-b913-8f305a3af254-kube-api-access-z4fhp\") pod \"maas-api-revoke-sub-75cdb93f-rmrrb\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:46.052753 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.052725 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75cdb93f-rmrrb\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:46.054659 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.054632 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75cdb93f-rmrrb\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:46.106368 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.106345 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4fhp\" (UniqueName: \"kubernetes.io/projected/9b71bea1-31a1-4e7b-b913-8f305a3af254-kube-api-access-z4fhp\") pod \"maas-api-revoke-sub-75cdb93f-rmrrb\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:46.122772 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.122717 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw"] Nov 02 23:12:46.126953 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.126933 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:46.175990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.175957 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:46.181539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.181517 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw"] Nov 02 23:12:46.254556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.254131 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwgpg\" (UniqueName: \"kubernetes.io/projected/f6a1a57c-d16d-4709-b751-53112ed20714-kube-api-access-kwgpg\") pod \"maas-api-revoke-sub-c6219c92-h7zfw\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:46.254556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.254326 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6219c92-h7zfw\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:46.254556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.254438 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6219c92-h7zfw\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:46.355724 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.355691 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6219c92-h7zfw\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:46.355912 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.355808 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgpg\" (UniqueName: \"kubernetes.io/projected/f6a1a57c-d16d-4709-b751-53112ed20714-kube-api-access-kwgpg\") pod \"maas-api-revoke-sub-c6219c92-h7zfw\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:46.355973 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.355928 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6219c92-h7zfw\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:46.356710 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.356659 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6219c92-h7zfw\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:46.358584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.358561 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6219c92-h7zfw\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:46.410758 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.410707 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgpg\" (UniqueName: \"kubernetes.io/projected/f6a1a57c-d16d-4709-b751-53112ed20714-kube-api-access-kwgpg\") pod \"maas-api-revoke-sub-c6219c92-h7zfw\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:46.444987 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.444962 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:46.559389 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.559358 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb"] Nov 02 23:12:46.559504 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:12:46.559388 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b71bea1_31a1_4e7b_b913_8f305a3af254.slice/crio-c8180e65dbd10391eead1b9c9b48bb0f1bb9b1e4b55df424a1f7b496ed0686d2 WatchSource:0}: Error finding container c8180e65dbd10391eead1b9c9b48bb0f1bb9b1e4b55df424a1f7b496ed0686d2: Status 404 returned error can't find the container with id c8180e65dbd10391eead1b9c9b48bb0f1bb9b1e4b55df424a1f7b496ed0686d2 Nov 02 23:12:46.704608 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.704534 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh"] Nov 02 23:12:46.710643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.710612 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.761922 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.761885 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.762078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.761931 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.762078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.761968 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.762078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.762010 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkb4p\" (UniqueName: \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-kube-api-access-fkb4p\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.762078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.762064 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.762320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.762095 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80910ec3-8b61-4304-a410-d4bca2b27fe7-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.762320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.762190 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.762320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.762303 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.762458 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.762377 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.766955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.766925 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh"] Nov 02 23:12:46.824114 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.824075 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw"] Nov 02 23:12:46.826719 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:12:46.826690 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a1a57c_d16d_4709_b751_53112ed20714.slice/crio-b4c5aec9fbb5244c6c22ce7a0f1254df82c2b3d6219679dfd58ff545baa64aac WatchSource:0}: Error finding container b4c5aec9fbb5244c6c22ce7a0f1254df82c2b3d6219679dfd58ff545baa64aac: Status 404 returned error can't find the container with id b4c5aec9fbb5244c6c22ce7a0f1254df82c2b3d6219679dfd58ff545baa64aac Nov 02 23:12:46.863586 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.863567 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.863604 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.863639 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.863822 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkb4p\" (UniqueName: \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-kube-api-access-fkb4p\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.863918 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.863956 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80910ec3-8b61-4304-a410-d4bca2b27fe7-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.864043 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.864122 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.864137 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.864240 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.864265 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864648 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.864410 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.864845 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.864822 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.865101 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.865059 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80910ec3-8b61-4304-a410-d4bca2b27fe7-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.866963 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.866936 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.867075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.867058 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.916969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.916943 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:46.917781 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:46.917764 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkb4p\" (UniqueName: \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-kube-api-access-fkb4p\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:47.025382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:47.025139 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:47.305931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:47.305844 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" event={"ID":"9b71bea1-31a1-4e7b-b913-8f305a3af254","Type":"ContainerStarted","Data":"1226f148683b7e04b0d81f9a200909868c4b628c5c77f0d3dcb9998b93045c32"} Nov 02 23:12:47.305931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:47.305905 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" event={"ID":"9b71bea1-31a1-4e7b-b913-8f305a3af254","Type":"ContainerStarted","Data":"c8180e65dbd10391eead1b9c9b48bb0f1bb9b1e4b55df424a1f7b496ed0686d2"} Nov 02 23:12:47.307978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:47.307944 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" event={"ID":"f6a1a57c-d16d-4709-b751-53112ed20714","Type":"ContainerStarted","Data":"23ab8e9ed3271b2484fdf2e9d83a62ae47862bba3a6b3f72fcbb25dd32cd92f4"} Nov 02 23:12:47.308123 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:47.307988 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" event={"ID":"f6a1a57c-d16d-4709-b751-53112ed20714","Type":"ContainerStarted","Data":"b4c5aec9fbb5244c6c22ce7a0f1254df82c2b3d6219679dfd58ff545baa64aac"} Nov 02 23:12:47.415126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:47.415073 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" podStartSLOduration=2.415060204 podStartE2EDuration="2.415060204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:47.413970053 +0000 UTC m=+1160.821281883" watchObservedRunningTime="2026-11-02 23:12:47.415060204 +0000 UTC m=+1160.822372031" Nov 02 23:12:47.447226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:47.447196 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh"] Nov 02 23:12:47.449810 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:12:47.449774 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80910ec3_8b61_4304_a410_d4bca2b27fe7.slice/crio-36d7ab01bde8cf72d7f058a2af0480bdc165681d7774d4357e3e992a17feef36 WatchSource:0}: Error finding container 36d7ab01bde8cf72d7f058a2af0480bdc165681d7774d4357e3e992a17feef36: Status 404 returned error can't find the container with id 36d7ab01bde8cf72d7f058a2af0480bdc165681d7774d4357e3e992a17feef36 Nov 02 23:12:47.452816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:47.452779 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:47.452931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:47.452842 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:47.452931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:47.452872 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:12:47.520873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:47.520827 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" podStartSLOduration=1.520814297 podStartE2EDuration="1.520814297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:47.519530672 +0000 UTC m=+1160.926842514" watchObservedRunningTime="2026-11-02 23:12:47.520814297 +0000 UTC m=+1160.928126133" Nov 02 23:12:48.320160 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:48.320124 5672 generic.go:356] "Generic (PLEG): container finished" podID="9b71bea1-31a1-4e7b-b913-8f305a3af254" containerID="1226f148683b7e04b0d81f9a200909868c4b628c5c77f0d3dcb9998b93045c32" exitCode=0 Nov 02 23:12:48.320649 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:48.320408 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" event={"ID":"9b71bea1-31a1-4e7b-b913-8f305a3af254","Type":"ContainerDied","Data":"1226f148683b7e04b0d81f9a200909868c4b628c5c77f0d3dcb9998b93045c32"} Nov 02 23:12:48.322926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:48.322901 5672 generic.go:356] "Generic (PLEG): container finished" podID="f6a1a57c-d16d-4709-b751-53112ed20714" containerID="23ab8e9ed3271b2484fdf2e9d83a62ae47862bba3a6b3f72fcbb25dd32cd92f4" exitCode=0 Nov 02 23:12:48.323170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:48.322990 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" event={"ID":"f6a1a57c-d16d-4709-b751-53112ed20714","Type":"ContainerDied","Data":"23ab8e9ed3271b2484fdf2e9d83a62ae47862bba3a6b3f72fcbb25dd32cd92f4"} Nov 02 23:12:48.325713 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:48.325675 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" event={"ID":"80910ec3-8b61-4304-a410-d4bca2b27fe7","Type":"ContainerStarted","Data":"0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34"} Nov 02 23:12:48.325902 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:48.325886 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" event={"ID":"80910ec3-8b61-4304-a410-d4bca2b27fe7","Type":"ContainerStarted","Data":"36d7ab01bde8cf72d7f058a2af0480bdc165681d7774d4357e3e992a17feef36"} Nov 02 23:12:48.668084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:48.667963 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podStartSLOduration=2.667943164 podStartE2EDuration="2.667943164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:48.560828169 +0000 UTC m=+1161.968140000" watchObservedRunningTime="2026-11-02 23:12:48.667943164 +0000 UTC m=+1162.075254994" Nov 02 23:12:49.049288 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.047660 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:49.055181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.049930 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:49.055181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.049989 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 23:12:49.612835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.612784 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:49.616676 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.616656 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:49.700655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.700627 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-openshift-ca\") pod \"9b71bea1-31a1-4e7b-b913-8f305a3af254\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " Nov 02 23:12:49.700861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.700848 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b71bea1-31a1-4e7b-b913-8f305a3af254-kube-api-access-z4fhp\" (UniqueName: \"kubernetes.io/projected/9b71bea1-31a1-4e7b-b913-8f305a3af254-kube-api-access-z4fhp\") pod \"9b71bea1-31a1-4e7b-b913-8f305a3af254\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " Nov 02 23:12:49.700941 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.700930 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-xks-ca\") pod \"f6a1a57c-d16d-4709-b751-53112ed20714\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " Nov 02 23:12:49.701058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.701049 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-xks-ca\") pod \"9b71bea1-31a1-4e7b-b913-8f305a3af254\" (UID: \"9b71bea1-31a1-4e7b-b913-8f305a3af254\") " Nov 02 23:12:49.701207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.700944 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-openshift-ca" pod "9b71bea1-31a1-4e7b-b913-8f305a3af254" (UID: "9b71bea1-31a1-4e7b-b913-8f305a3af254"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:49.701282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.701191 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-openshift-ca\") pod \"f6a1a57c-d16d-4709-b751-53112ed20714\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " Nov 02 23:12:49.701337 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.701310 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a1a57c-d16d-4709-b751-53112ed20714-kube-api-access-kwgpg\" (UniqueName: \"kubernetes.io/projected/f6a1a57c-d16d-4709-b751-53112ed20714-kube-api-access-kwgpg\") pod \"f6a1a57c-d16d-4709-b751-53112ed20714\" (UID: \"f6a1a57c-d16d-4709-b751-53112ed20714\") " Nov 02 23:12:49.701528 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.701508 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-openshift-ca" pod "f6a1a57c-d16d-4709-b751-53112ed20714" (UID: "f6a1a57c-d16d-4709-b751-53112ed20714"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:12:49.701884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.701861 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:49.701960 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.701889 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:49.703316 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.703293 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-xks-ca" pod "f6a1a57c-d16d-4709-b751-53112ed20714" (UID: "f6a1a57c-d16d-4709-b751-53112ed20714"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:49.703419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.703348 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b71bea1-31a1-4e7b-b913-8f305a3af254-kube-api-access-z4fhp" pod "9b71bea1-31a1-4e7b-b913-8f305a3af254" (UID: "9b71bea1-31a1-4e7b-b913-8f305a3af254"). InnerVolumeSpecName "kube-api-access-z4fhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:49.703419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.703412 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a1a57c-d16d-4709-b751-53112ed20714-kube-api-access-kwgpg" pod "f6a1a57c-d16d-4709-b751-53112ed20714" (UID: "f6a1a57c-d16d-4709-b751-53112ed20714"). InnerVolumeSpecName "kube-api-access-kwgpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:12:49.703536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.703501 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-xks-ca" pod "9b71bea1-31a1-4e7b-b913-8f305a3af254" (UID: "9b71bea1-31a1-4e7b-b913-8f305a3af254"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:12:49.804034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.804002 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4fhp\" (UniqueName: \"kubernetes.io/projected/9b71bea1-31a1-4e7b-b913-8f305a3af254-kube-api-access-z4fhp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:49.804034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.804034 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a1a57c-d16d-4709-b751-53112ed20714-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:49.804286 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.804045 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b71bea1-31a1-4e7b-b913-8f305a3af254-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:49.804286 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:49.804055 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwgpg\" (UniqueName: \"kubernetes.io/projected/f6a1a57c-d16d-4709-b751-53112ed20714-kube-api-access-kwgpg\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:12:50.050174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:50.049828 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:50.050174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:50.049885 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 23:12:50.368228 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:50.367764 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" Nov 02 23:12:50.368228 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:50.367801 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw" event={"ID":"f6a1a57c-d16d-4709-b751-53112ed20714","Type":"ContainerDied","Data":"b4c5aec9fbb5244c6c22ce7a0f1254df82c2b3d6219679dfd58ff545baa64aac"} Nov 02 23:12:50.368228 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:50.367880 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4c5aec9fbb5244c6c22ce7a0f1254df82c2b3d6219679dfd58ff545baa64aac" Nov 02 23:12:50.371938 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:50.371888 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" Nov 02 23:12:50.372127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:50.372104 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb" event={"ID":"9b71bea1-31a1-4e7b-b913-8f305a3af254","Type":"ContainerDied","Data":"c8180e65dbd10391eead1b9c9b48bb0f1bb9b1e4b55df424a1f7b496ed0686d2"} Nov 02 23:12:50.372350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:50.372321 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8180e65dbd10391eead1b9c9b48bb0f1bb9b1e4b55df424a1f7b496ed0686d2" Nov 02 23:12:50.955091 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:50.955046 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:12:51.045173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:51.039565 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:51.045173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:51.039627 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 23:12:51.051531 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:51.051490 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w"] Nov 02 23:12:51.057193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:51.057165 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1c46241-jzv5w"] Nov 02 23:12:52.037176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:52.033537 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:52.037176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:52.033601 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 23:12:52.755193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:52.752803 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacbcf11-86bd-4805-bef1-49fbf14451ff" path="/var/lib/kubelet/pods/aacbcf11-86bd-4805-bef1-49fbf14451ff/volumes" Nov 02 23:12:53.036230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.032592 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:53.036230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.032655 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 23:12:53.151821 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.151258 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:53.318988 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.318263 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:53.355185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.350234 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 23:12:53.372004 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.371938 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 23:12:53.434636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.433834 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 23:12:53.498607 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.498565 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:53.629327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.629222 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:53.652279 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.652231 5672 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:53.713291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.713239 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:53.766302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.766243 5672 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:53.793173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:53.784237 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:54.039173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.038558 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:54.039173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.038624 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 23:12:54.137549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.137379 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:12:54.137549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.137462 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:54.154362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.152276 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:54.154362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.152715 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:12:54.154362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.154232 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:12:54.154362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.154281 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:54.319047 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.318950 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:54.319695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.319667 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:12:54.338550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.338420 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:12:54.338550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.338480 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:54.375174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.350627 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:54.375174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.352256 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:12:54.375174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.373641 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:54.375174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.374030 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:12:54.451174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.439505 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 23:12:54.451174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.439929 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:12:54.500323 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.500283 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:54.555934 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.555864 5672 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:54.577119 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.576862 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:12:54.632184 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.631414 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:54.632184 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.631790 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:12:54.888747 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:54.873505 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 23:12:55.020451 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.020399 5672 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:55.106342 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.038143 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:55.106606 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.106572 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 23:12:55.222231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.216686 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:55.234198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.231537 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:12:55.370170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.347463 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:55.370170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.365677 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 23:12:55.395171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.381792 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:55.478372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.449333 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:55.700172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.700113 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:55.702454 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.702399 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:55.880227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.877386 5672 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:55.880227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.877864 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:55.880227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:55.878142 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:56.034733 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.034693 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:56.035070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.035037 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 23:12:56.237259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.233721 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:12:56.237259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.235214 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:56.413216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.379288 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:56.413216 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:12:56.381388 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:12:56.413216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.381938 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:56.415433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.414688 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:56.709481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.709367 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 23:12:56.727328 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.712312 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:56.880628 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.880232 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:56.915065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.908573 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:12:56.915065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.908641 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:12:56.915065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.908678 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:12:56.915065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.908711 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:12:56.938335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.938305 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:12:56.939051 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.939022 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:12:56.942659 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.942640 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:12:56.991226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.991195 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp"] Nov 02 23:12:56.991663 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.991650 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f6a1a57c-d16d-4709-b751-53112ed20714" containerName="revoke-keys" Nov 02 23:12:56.991706 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.991663 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a1a57c-d16d-4709-b751-53112ed20714" containerName="revoke-keys" Nov 02 23:12:56.991738 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.991708 5672 cpu_manager.go:414] "containerMap: removing container" podUID="9b71bea1-31a1-4e7b-b913-8f305a3af254" containerName="revoke-keys" Nov 02 23:12:56.991738 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.991713 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b71bea1-31a1-4e7b-b913-8f305a3af254" containerName="revoke-keys" Nov 02 23:12:56.991803 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.991784 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a1a57c-d16d-4709-b751-53112ed20714" containerName="revoke-keys" Nov 02 23:12:56.991803 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.991796 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b71bea1-31a1-4e7b-b913-8f305a3af254" containerName="revoke-keys" Nov 02 23:12:56.996532 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:56.996511 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:12:57.025532 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.025498 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 23:12:57.025723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.025553 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 23:12:57.045598 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.045565 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:12:57.045804 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.045769 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:12:57.045900 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.045825 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:12:57.045900 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.045885 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:12:57.050055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.050020 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp"] Nov 02 23:12:57.163992 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.163953 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7403b508-shqxp\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:12:57.164181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.164073 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7403b508-shqxp\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:12:57.164276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.164255 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cstl4\" (UniqueName: \"kubernetes.io/projected/32c17e66-5390-4350-acc8-007d6a6192ea-kube-api-access-cstl4\") pod \"maas-api-revoke-sub-7403b508-shqxp\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:12:57.265214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.265110 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7403b508-shqxp\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:12:57.265214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.265181 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7403b508-shqxp\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:12:57.265735 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.265280 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cstl4\" (UniqueName: \"kubernetes.io/projected/32c17e66-5390-4350-acc8-007d6a6192ea-kube-api-access-cstl4\") pod \"maas-api-revoke-sub-7403b508-shqxp\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:12:57.266819 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.266793 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7403b508-shqxp\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:12:57.268594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.268488 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7403b508-shqxp\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:12:57.319072 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.319030 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cstl4\" (UniqueName: \"kubernetes.io/projected/32c17e66-5390-4350-acc8-007d6a6192ea-kube-api-access-cstl4\") pod \"maas-api-revoke-sub-7403b508-shqxp\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:12:57.613537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:57.613459 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:12:58.052854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:58.052719 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp"] Nov 02 23:12:58.055318 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:12:58.055282 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c17e66_5390_4350_acc8_007d6a6192ea.slice/crio-e95cf8037a6866b095967c4462088e7174d76265b2590f3a4db76f426790b73c WatchSource:0}: Error finding container e95cf8037a6866b095967c4462088e7174d76265b2590f3a4db76f426790b73c: Status 404 returned error can't find the container with id e95cf8037a6866b095967c4462088e7174d76265b2590f3a4db76f426790b73c Nov 02 23:12:58.729203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:58.728635 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" event={"ID":"32c17e66-5390-4350-acc8-007d6a6192ea","Type":"ContainerStarted","Data":"ea7580fd38cd5b8f1ff9f99edafcaa1b319e9366cbb1e9c345603f3eac2dcbdb"} Nov 02 23:12:58.729203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:58.728695 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" event={"ID":"32c17e66-5390-4350-acc8-007d6a6192ea","Type":"ContainerStarted","Data":"e95cf8037a6866b095967c4462088e7174d76265b2590f3a4db76f426790b73c"} Nov 02 23:12:58.838345 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:58.838264 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" podStartSLOduration=2.838244863 podStartE2EDuration="2.838244863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:58.833799552 +0000 UTC m=+1172.241111383" watchObservedRunningTime="2026-11-02 23:12:58.838244863 +0000 UTC m=+1172.245556694" Nov 02 23:12:59.067395 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:59.067300 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:12:59.067659 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:59.067623 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:12:59.364693 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:59.364610 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:59.364979 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:59.364933 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:59.365823 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:59.365800 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:12:59.738103 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:59.738068 5672 generic.go:356] "Generic (PLEG): container finished" podID="32c17e66-5390-4350-acc8-007d6a6192ea" containerID="ea7580fd38cd5b8f1ff9f99edafcaa1b319e9366cbb1e9c345603f3eac2dcbdb" exitCode=0 Nov 02 23:12:59.738613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:12:59.738169 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" event={"ID":"32c17e66-5390-4350-acc8-007d6a6192ea","Type":"ContainerDied","Data":"ea7580fd38cd5b8f1ff9f99edafcaa1b319e9366cbb1e9c345603f3eac2dcbdb"} Nov 02 23:13:00.902397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:00.902372 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:13:00.912848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:00.912819 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32c17e66-5390-4350-acc8-007d6a6192ea-kube-api-access-cstl4\" (UniqueName: \"kubernetes.io/projected/32c17e66-5390-4350-acc8-007d6a6192ea-kube-api-access-cstl4\") pod \"32c17e66-5390-4350-acc8-007d6a6192ea\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " Nov 02 23:13:00.912975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:00.912866 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-xks-ca\") pod \"32c17e66-5390-4350-acc8-007d6a6192ea\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " Nov 02 23:13:00.913053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:00.913006 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-openshift-ca\") pod \"32c17e66-5390-4350-acc8-007d6a6192ea\" (UID: \"32c17e66-5390-4350-acc8-007d6a6192ea\") " Nov 02 23:13:00.913454 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:00.913421 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-openshift-ca" pod "32c17e66-5390-4350-acc8-007d6a6192ea" (UID: "32c17e66-5390-4350-acc8-007d6a6192ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:13:00.913568 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:00.913538 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:00.914944 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:00.914917 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c17e66-5390-4350-acc8-007d6a6192ea-kube-api-access-cstl4" pod "32c17e66-5390-4350-acc8-007d6a6192ea" (UID: "32c17e66-5390-4350-acc8-007d6a6192ea"). InnerVolumeSpecName "kube-api-access-cstl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:13:00.915438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:00.915416 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-xks-ca" pod "32c17e66-5390-4350-acc8-007d6a6192ea" (UID: "32c17e66-5390-4350-acc8-007d6a6192ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:13:01.015240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.014908 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cstl4\" (UniqueName: \"kubernetes.io/projected/32c17e66-5390-4350-acc8-007d6a6192ea-kube-api-access-cstl4\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:01.015240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.014947 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c17e66-5390-4350-acc8-007d6a6192ea-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:01.593902 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.593867 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g"] Nov 02 23:13:01.594444 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.594429 5672 cpu_manager.go:414] "containerMap: removing container" podUID="32c17e66-5390-4350-acc8-007d6a6192ea" containerName="revoke-keys" Nov 02 23:13:01.594495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.594445 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32c17e66-5390-4350-acc8-007d6a6192ea" containerName="revoke-keys" Nov 02 23:13:01.594571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.594561 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="32c17e66-5390-4350-acc8-007d6a6192ea" containerName="revoke-keys" Nov 02 23:13:01.598266 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.598244 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:01.623289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.623260 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57nbl\" (UniqueName: \"kubernetes.io/projected/9c972f72-b716-434a-a2fa-49cfcd77aac3-kube-api-access-57nbl\") pod \"maas-api-revoke-sub-f5066a43-ngl2g\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:01.623404 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.623388 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5066a43-ngl2g\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:01.623537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.623523 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5066a43-ngl2g\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:01.651665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.651631 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g"] Nov 02 23:13:01.724656 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.724624 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5066a43-ngl2g\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:01.724812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.724687 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57nbl\" (UniqueName: \"kubernetes.io/projected/9c972f72-b716-434a-a2fa-49cfcd77aac3-kube-api-access-57nbl\") pod \"maas-api-revoke-sub-f5066a43-ngl2g\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:01.724812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.724791 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5066a43-ngl2g\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:01.725411 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.725388 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5066a43-ngl2g\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:01.727191 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.727164 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5066a43-ngl2g\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:01.753109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.753083 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" Nov 02 23:13:01.753237 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.753112 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp" event={"ID":"32c17e66-5390-4350-acc8-007d6a6192ea","Type":"ContainerDied","Data":"e95cf8037a6866b095967c4462088e7174d76265b2590f3a4db76f426790b73c"} Nov 02 23:13:01.753237 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.753172 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e95cf8037a6866b095967c4462088e7174d76265b2590f3a4db76f426790b73c" Nov 02 23:13:01.787238 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.787217 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57nbl\" (UniqueName: \"kubernetes.io/projected/9c972f72-b716-434a-a2fa-49cfcd77aac3-kube-api-access-57nbl\") pod \"maas-api-revoke-sub-f5066a43-ngl2g\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:01.912289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:01.912221 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:02.282216 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:13:02.282167 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c972f72_b716_434a_a2fa_49cfcd77aac3.slice/crio-878c4417195d19a122df547eb3181ae63bbb613f03af5ffd4ff87a0dcfc02ae1 WatchSource:0}: Error finding container 878c4417195d19a122df547eb3181ae63bbb613f03af5ffd4ff87a0dcfc02ae1: Status 404 returned error can't find the container with id 878c4417195d19a122df547eb3181ae63bbb613f03af5ffd4ff87a0dcfc02ae1 Nov 02 23:13:02.283337 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:02.283309 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g"] Nov 02 23:13:02.761928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:02.761896 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" event={"ID":"9c972f72-b716-434a-a2fa-49cfcd77aac3","Type":"ContainerStarted","Data":"7cc065ff565b4f01f7b0abe3d2dcbadb774e42fb80f13e6355675f07f7d0b25b"} Nov 02 23:13:02.762112 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:02.761949 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" event={"ID":"9c972f72-b716-434a-a2fa-49cfcd77aac3","Type":"ContainerStarted","Data":"878c4417195d19a122df547eb3181ae63bbb613f03af5ffd4ff87a0dcfc02ae1"} Nov 02 23:13:02.873069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:02.872963 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" podStartSLOduration=1.87294889 podStartE2EDuration="1.87294889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:02.870116837 +0000 UTC m=+1176.277428669" watchObservedRunningTime="2026-11-02 23:13:02.87294889 +0000 UTC m=+1176.280260720" Nov 02 23:13:03.776862 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:03.776827 5672 generic.go:356] "Generic (PLEG): container finished" podID="9c972f72-b716-434a-a2fa-49cfcd77aac3" containerID="7cc065ff565b4f01f7b0abe3d2dcbadb774e42fb80f13e6355675f07f7d0b25b" exitCode=0 Nov 02 23:13:03.777268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:03.776912 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" event={"ID":"9c972f72-b716-434a-a2fa-49cfcd77aac3","Type":"ContainerDied","Data":"7cc065ff565b4f01f7b0abe3d2dcbadb774e42fb80f13e6355675f07f7d0b25b"} Nov 02 23:13:04.944855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:04.944827 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:05.061448 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.061365 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-openshift-ca\") pod \"9c972f72-b716-434a-a2fa-49cfcd77aac3\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " Nov 02 23:13:05.061613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.061533 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-xks-ca\") pod \"9c972f72-b716-434a-a2fa-49cfcd77aac3\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " Nov 02 23:13:05.061613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.061594 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c972f72-b716-434a-a2fa-49cfcd77aac3-kube-api-access-57nbl\" (UniqueName: \"kubernetes.io/projected/9c972f72-b716-434a-a2fa-49cfcd77aac3-kube-api-access-57nbl\") pod \"9c972f72-b716-434a-a2fa-49cfcd77aac3\" (UID: \"9c972f72-b716-434a-a2fa-49cfcd77aac3\") " Nov 02 23:13:05.061805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.061776 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-openshift-ca" pod "9c972f72-b716-434a-a2fa-49cfcd77aac3" (UID: "9c972f72-b716-434a-a2fa-49cfcd77aac3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:13:05.062168 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.062129 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:05.064425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.064393 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-xks-ca" pod "9c972f72-b716-434a-a2fa-49cfcd77aac3" (UID: "9c972f72-b716-434a-a2fa-49cfcd77aac3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:13:05.064425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.064399 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c972f72-b716-434a-a2fa-49cfcd77aac3-kube-api-access-57nbl" pod "9c972f72-b716-434a-a2fa-49cfcd77aac3" (UID: "9c972f72-b716-434a-a2fa-49cfcd77aac3"). InnerVolumeSpecName "kube-api-access-57nbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:13:05.164076 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.164038 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c972f72-b716-434a-a2fa-49cfcd77aac3-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:05.164076 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.164079 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57nbl\" (UniqueName: \"kubernetes.io/projected/9c972f72-b716-434a-a2fa-49cfcd77aac3-kube-api-access-57nbl\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:05.790478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.790394 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" Nov 02 23:13:05.790478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.790422 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g" event={"ID":"9c972f72-b716-434a-a2fa-49cfcd77aac3","Type":"ContainerDied","Data":"878c4417195d19a122df547eb3181ae63bbb613f03af5ffd4ff87a0dcfc02ae1"} Nov 02 23:13:05.790478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:05.790471 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="878c4417195d19a122df547eb3181ae63bbb613f03af5ffd4ff87a0dcfc02ae1" Nov 02 23:13:08.309552 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.309507 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj"] Nov 02 23:13:08.310309 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.310282 5672 cpu_manager.go:414] "containerMap: removing container" podUID="9c972f72-b716-434a-a2fa-49cfcd77aac3" containerName="revoke-keys" Nov 02 23:13:08.310309 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.310309 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c972f72-b716-434a-a2fa-49cfcd77aac3" containerName="revoke-keys" Nov 02 23:13:08.310476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.310464 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c972f72-b716-434a-a2fa-49cfcd77aac3" containerName="revoke-keys" Nov 02 23:13:08.314361 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.314338 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:08.365733 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.365704 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj"] Nov 02 23:13:08.399821 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.399786 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5ed25c8-84gpj\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:08.399990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.399852 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5ed25c8-84gpj\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:08.400059 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.400044 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fjm\" (UniqueName: \"kubernetes.io/projected/3b26163c-1466-453b-9685-eaace6d1bfbe-kube-api-access-76fjm\") pod \"maas-api-revoke-sub-a5ed25c8-84gpj\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:08.408164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.408119 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:13:08.408276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.408185 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:13:08.408276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.408209 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:13:08.409137 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.409122 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:13:08.501395 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.501348 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5ed25c8-84gpj\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:08.501550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.501417 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5ed25c8-84gpj\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:08.501550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.501534 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fjm\" (UniqueName: \"kubernetes.io/projected/3b26163c-1466-453b-9685-eaace6d1bfbe-kube-api-access-76fjm\") pod \"maas-api-revoke-sub-a5ed25c8-84gpj\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:08.501982 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.501963 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5ed25c8-84gpj\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:08.503715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.503695 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5ed25c8-84gpj\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:08.554440 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.554409 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fjm\" (UniqueName: \"kubernetes.io/projected/3b26163c-1466-453b-9685-eaace6d1bfbe-kube-api-access-76fjm\") pod \"maas-api-revoke-sub-a5ed25c8-84gpj\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:08.627525 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.627445 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:08.994968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:08.994945 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj"] Nov 02 23:13:08.995518 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:13:08.995487 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b26163c_1466_453b_9685_eaace6d1bfbe.slice/crio-c7886a066fc2fbe9678b8be1a168afadbd6533ffd650f19de96d6e14b1ee6972 WatchSource:0}: Error finding container c7886a066fc2fbe9678b8be1a168afadbd6533ffd650f19de96d6e14b1ee6972: Status 404 returned error can't find the container with id c7886a066fc2fbe9678b8be1a168afadbd6533ffd650f19de96d6e14b1ee6972 Nov 02 23:13:09.822310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:09.822267 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" event={"ID":"3b26163c-1466-453b-9685-eaace6d1bfbe","Type":"ContainerStarted","Data":"095e9e5fd05fff087a453324ba4292269115a4ec84823de98aa2a48ba0e73c44"} Nov 02 23:13:09.822775 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:09.822335 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" event={"ID":"3b26163c-1466-453b-9685-eaace6d1bfbe","Type":"ContainerStarted","Data":"c7886a066fc2fbe9678b8be1a168afadbd6533ffd650f19de96d6e14b1ee6972"} Nov 02 23:13:09.930074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:09.930024 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" podStartSLOduration=1.930010948 podStartE2EDuration="1.930010948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:09.928249572 +0000 UTC m=+1183.335561401" watchObservedRunningTime="2026-11-02 23:13:09.930010948 +0000 UTC m=+1183.337322778" Nov 02 23:13:10.837805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:10.837770 5672 generic.go:356] "Generic (PLEG): container finished" podID="3b26163c-1466-453b-9685-eaace6d1bfbe" containerID="095e9e5fd05fff087a453324ba4292269115a4ec84823de98aa2a48ba0e73c44" exitCode=0 Nov 02 23:13:10.838920 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:10.838869 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" event={"ID":"3b26163c-1466-453b-9685-eaace6d1bfbe","Type":"ContainerDied","Data":"095e9e5fd05fff087a453324ba4292269115a4ec84823de98aa2a48ba0e73c44"} Nov 02 23:13:12.026444 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.026417 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:12.053751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.053719 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-openshift-ca\") pod \"3b26163c-1466-453b-9685-eaace6d1bfbe\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " Nov 02 23:13:12.053915 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.053798 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b26163c-1466-453b-9685-eaace6d1bfbe-kube-api-access-76fjm\" (UniqueName: \"kubernetes.io/projected/3b26163c-1466-453b-9685-eaace6d1bfbe-kube-api-access-76fjm\") pod \"3b26163c-1466-453b-9685-eaace6d1bfbe\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " Nov 02 23:13:12.053969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.053926 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-xks-ca\") pod \"3b26163c-1466-453b-9685-eaace6d1bfbe\" (UID: \"3b26163c-1466-453b-9685-eaace6d1bfbe\") " Nov 02 23:13:12.054141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.054114 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-openshift-ca" pod "3b26163c-1466-453b-9685-eaace6d1bfbe" (UID: "3b26163c-1466-453b-9685-eaace6d1bfbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:13:12.054501 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.054471 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:12.056234 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.056210 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-xks-ca" pod "3b26163c-1466-453b-9685-eaace6d1bfbe" (UID: "3b26163c-1466-453b-9685-eaace6d1bfbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:13:12.056333 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.056218 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b26163c-1466-453b-9685-eaace6d1bfbe-kube-api-access-76fjm" pod "3b26163c-1466-453b-9685-eaace6d1bfbe" (UID: "3b26163c-1466-453b-9685-eaace6d1bfbe"). InnerVolumeSpecName "kube-api-access-76fjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:13:12.165249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.165118 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76fjm\" (UniqueName: \"kubernetes.io/projected/3b26163c-1466-453b-9685-eaace6d1bfbe-kube-api-access-76fjm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:12.165249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.165174 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b26163c-1466-453b-9685-eaace6d1bfbe-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:12.218843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.218364 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh"] Nov 02 23:13:12.219030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.218865 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3b26163c-1466-453b-9685-eaace6d1bfbe" containerName="revoke-keys" Nov 02 23:13:12.219030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.218879 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b26163c-1466-453b-9685-eaace6d1bfbe" containerName="revoke-keys" Nov 02 23:13:12.219030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.219001 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b26163c-1466-453b-9685-eaace6d1bfbe" containerName="revoke-keys" Nov 02 23:13:12.223016 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.222988 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.266899 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.266865 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.267079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.266941 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.267079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.266969 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.267079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.267032 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.267217 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.267198 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.267257 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.267230 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.267296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.267262 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.267438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.267415 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hfct\" (UniqueName: \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-kube-api-access-5hfct\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.267528 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.267512 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.329628 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.329591 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh"] Nov 02 23:13:12.334894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.334867 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j"] Nov 02 23:13:12.339768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.339745 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:12.369025 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.368991 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.369209 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.369092 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.369209 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.369133 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.369209 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.369176 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.369402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.369311 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.369402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.369355 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.369402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.369386 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.369565 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.369445 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.369565 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.369474 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfct\" (UniqueName: \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-kube-api-access-5hfct\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.370138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.370084 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.370274 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.370173 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.370635 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.370613 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.370817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.370794 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.370888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.370876 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.372639 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.372616 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.374947 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.374921 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.389131 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.389103 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j"] Nov 02 23:13:12.443251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.431740 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.443251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.434960 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hfct\" (UniqueName: \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-kube-api-access-5hfct\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.471397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.471360 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-217eb132-mhg7j\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:12.471595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.471444 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-217eb132-mhg7j\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:12.471662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.471625 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbh2\" (UniqueName: \"kubernetes.io/projected/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-kube-api-access-tbbh2\") pod \"maas-api-revoke-sub-217eb132-mhg7j\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:12.544733 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.543380 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:12.573481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.573447 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-217eb132-mhg7j\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:12.573624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.573524 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-217eb132-mhg7j\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:12.573624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.573599 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbh2\" (UniqueName: \"kubernetes.io/projected/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-kube-api-access-tbbh2\") pod \"maas-api-revoke-sub-217eb132-mhg7j\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:12.574397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.574376 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-217eb132-mhg7j\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:12.576833 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.576760 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-217eb132-mhg7j\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:12.647892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.647467 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbh2\" (UniqueName: \"kubernetes.io/projected/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-kube-api-access-tbbh2\") pod \"maas-api-revoke-sub-217eb132-mhg7j\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:12.661405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.661370 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:12.896872 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.896828 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" event={"ID":"3b26163c-1466-453b-9685-eaace6d1bfbe","Type":"ContainerDied","Data":"c7886a066fc2fbe9678b8be1a168afadbd6533ffd650f19de96d6e14b1ee6972"} Nov 02 23:13:12.897052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.896932 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7886a066fc2fbe9678b8be1a168afadbd6533ffd650f19de96d6e14b1ee6972" Nov 02 23:13:12.897230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.897212 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj" Nov 02 23:13:12.995524 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.995476 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh"] Nov 02 23:13:12.995653 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:13:12.995536 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6ebfb5_2a91_4f51_aef7_205771fb098f.slice/crio-3221c95cbf37cfab78095cd1fe50abf6c18397f4175389961098f29dd36906c4 WatchSource:0}: Error finding container 3221c95cbf37cfab78095cd1fe50abf6c18397f4175389961098f29dd36906c4: Status 404 returned error can't find the container with id 3221c95cbf37cfab78095cd1fe50abf6c18397f4175389961098f29dd36906c4 Nov 02 23:13:12.998748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.998714 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:13:12.998850 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.998792 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:13:12.998850 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:12.998842 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:13:13.089726 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:13.089694 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j"] Nov 02 23:13:13.967998 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:13.966926 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" event={"ID":"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f","Type":"ContainerStarted","Data":"b67907cb5e15ed4fb55dd502f59be4b73134934ca0e203741e76d8b0951a4f2a"} Nov 02 23:13:13.967998 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:13.966995 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" event={"ID":"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f","Type":"ContainerStarted","Data":"42573dbda914472ed52c1db3c131e0ebf3e95a4fffd099a637ec44532363321b"} Nov 02 23:13:13.971455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:13.970704 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" event={"ID":"2d6ebfb5-2a91-4f51-aef7-205771fb098f","Type":"ContainerStarted","Data":"729ad55d3e047625dcb0d3c4c005c735875807d46630f601f2f195239453a3c1"} Nov 02 23:13:13.971455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:13.970758 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" event={"ID":"2d6ebfb5-2a91-4f51-aef7-205771fb098f","Type":"ContainerStarted","Data":"3221c95cbf37cfab78095cd1fe50abf6c18397f4175389961098f29dd36906c4"} Nov 02 23:13:14.082465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:14.082413 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" podStartSLOduration=2.082397121 podStartE2EDuration="2.082397121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:14.080381755 +0000 UTC m=+1187.487693588" watchObservedRunningTime="2026-11-02 23:13:14.082397121 +0000 UTC m=+1187.489708964" Nov 02 23:13:14.201725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:14.199076 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" podStartSLOduration=2.199055674 podStartE2EDuration="2.199055674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:14.19692447 +0000 UTC m=+1187.604236300" watchObservedRunningTime="2026-11-02 23:13:14.199055674 +0000 UTC m=+1187.606367707" Nov 02 23:13:14.549855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:14.549769 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:14.550182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:14.550141 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:13:14.550357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:14.550329 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:13:14.997888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:14.997733 5672 generic.go:356] "Generic (PLEG): container finished" podID="3d6ee286-bb1c-42d5-a5ea-800f8de9d05f" containerID="b67907cb5e15ed4fb55dd502f59be4b73134934ca0e203741e76d8b0951a4f2a" exitCode=0 Nov 02 23:13:14.997888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:14.997809 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" event={"ID":"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f","Type":"ContainerDied","Data":"b67907cb5e15ed4fb55dd502f59be4b73134934ca0e203741e76d8b0951a4f2a"} Nov 02 23:13:15.548176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:15.545535 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:13:15.548176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:15.545601 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:13:16.332855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.332521 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:16.416904 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.416865 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-openshift-ca\") pod \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " Nov 02 23:13:16.416904 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.416907 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-xks-ca\") pod \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " Nov 02 23:13:16.417183 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.416946 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-kube-api-access-tbbh2\" (UniqueName: \"kubernetes.io/projected/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-kube-api-access-tbbh2\") pod \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\" (UID: \"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f\") " Nov 02 23:13:16.417269 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.417244 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-openshift-ca" pod "3d6ee286-bb1c-42d5-a5ea-800f8de9d05f" (UID: "3d6ee286-bb1c-42d5-a5ea-800f8de9d05f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:13:16.417623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.417602 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:16.420083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.419408 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-xks-ca" pod "3d6ee286-bb1c-42d5-a5ea-800f8de9d05f" (UID: "3d6ee286-bb1c-42d5-a5ea-800f8de9d05f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:13:16.420083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.419502 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-kube-api-access-tbbh2" pod "3d6ee286-bb1c-42d5-a5ea-800f8de9d05f" (UID: "3d6ee286-bb1c-42d5-a5ea-800f8de9d05f"). InnerVolumeSpecName "kube-api-access-tbbh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:13:16.537183 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.535205 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:16.537183 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.535240 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbh2\" (UniqueName: \"kubernetes.io/projected/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f-kube-api-access-tbbh2\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:16.547600 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.547570 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:13:16.552238 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:16.547808 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:13:17.098227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:17.098191 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" Nov 02 23:13:17.098423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:17.098199 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j" event={"ID":"3d6ee286-bb1c-42d5-a5ea-800f8de9d05f","Type":"ContainerDied","Data":"42573dbda914472ed52c1db3c131e0ebf3e95a4fffd099a637ec44532363321b"} Nov 02 23:13:17.098423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:17.098371 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42573dbda914472ed52c1db3c131e0ebf3e95a4fffd099a637ec44532363321b" Nov 02 23:13:17.545172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:17.544330 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:13:17.545172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:17.544393 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:13:18.562170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:18.549522 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:13:18.562170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:18.549583 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:13:19.546173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:19.545529 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 23:13:19.546173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:19.545603 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 23:13:21.067197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:21.060102 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv"] Nov 02 23:13:21.108420 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:21.108386 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbfee3a3-657fv"] Nov 02 23:13:21.460229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:21.459322 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:21.608174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:21.607268 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:21.608174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:21.607335 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:21.711227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:21.710161 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 23:13:21.795172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:21.794246 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 23:13:21.795172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:21.794681 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 02 23:13:21.912169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:21.900240 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:21.912169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:21.911687 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:22.235172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.230086 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:22.272178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.268206 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 23:13:22.392306 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.384527 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:22.651230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.630262 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:22.651230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.630357 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:22.651230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.640225 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:22.651230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.640702 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:13:22.729173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.723290 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 23:13:22.752175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.749773 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:22.752175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.749837 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:22.809425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.809386 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0988e692-a3ab-4478-a116-7a50b50113f1" path="/var/lib/kubelet/pods/0988e692-a3ab-4478-a116-7a50b50113f1/volumes" Nov 02 23:13:22.846074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.846044 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:13:22.857453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.823224 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 02 23:13:22.865115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.865089 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:13:22.866609 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.823106 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 23:13:22.867121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.867105 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:13:22.984870 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.984823 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:22.985494 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.985466 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 23:13:23.013190 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:22.986107 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:13:23.013190 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.010970 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:13:23.013190 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.011117 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:13:23.256242 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.239654 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:23.460170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.390050 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 23:13:23.460635 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.445926 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:13:23.460861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.460843 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:13:23.665109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.665016 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 23:13:23.917227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.809926 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:13:23.917227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.864230 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:23.917227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.812676 5672 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:23.917227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.819655 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:23.917227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.889255 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 02 23:13:23.960204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:23.946605 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:13:24.010493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.010453 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 02 23:13:24.011931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.011898 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:24.557009 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.530246 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:24.557009 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.551309 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:24.557009 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.552932 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:24.557009 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.554879 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:24.584194 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.562369 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:13:24.626947 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.626917 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:13:24.681936 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.681905 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:13:24.810574 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.810035 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:13:24.810751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.810599 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:13:24.812077 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.812047 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 02 23:13:24.815478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:24.815458 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:13:26.024238 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:13:26.024201 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:13:27.070852 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:27.070815 5672 scope.go:122] "RemoveContainer" containerID="fc1a8c3e5118bd512d43154ecbe57c5b2718b555fa580472eb3ae3c798d556d0" Nov 02 23:13:27.083912 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:27.083847 5672 scope.go:122] "RemoveContainer" containerID="955bf124f9ccac4a82a3d6b16f4878bb8a21217f9e3e934734f08e394a9c0faa" Nov 02 23:13:27.093130 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:27.093110 5672 scope.go:122] "RemoveContainer" containerID="db90600607de556857fd500bab8397037d6837e4c1f7d118333525346dc4d0cb" Nov 02 23:13:27.103265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:27.103246 5672 scope.go:122] "RemoveContainer" containerID="3d40acf99988a7d3892891ad7b447679d9ac4358c8df7f650fc174a7d15dfaa0" Nov 02 23:13:36.551270 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.551239 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72"] Nov 02 23:13:36.551778 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.551765 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3d6ee286-bb1c-42d5-a5ea-800f8de9d05f" containerName="revoke-keys" Nov 02 23:13:36.551823 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.551780 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6ee286-bb1c-42d5-a5ea-800f8de9d05f" containerName="revoke-keys" Nov 02 23:13:36.551886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.551877 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6ee286-bb1c-42d5-a5ea-800f8de9d05f" containerName="revoke-keys" Nov 02 23:13:36.557624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.557606 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:36.609159 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.609125 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72"] Nov 02 23:13:36.650704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.650674 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:13:36.650907 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.650693 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:13:36.651535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.651512 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:13:36.653141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.653119 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:13:36.654271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.654248 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb4e60f-8xx72\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:36.654405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.654386 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb4e60f-8xx72\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:36.654492 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.654474 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9xkq\" (UniqueName: \"kubernetes.io/projected/d7d50736-194e-42c4-b972-d0407bb2fe6d-kube-api-access-l9xkq\") pod \"maas-api-revoke-sub-3fb4e60f-8xx72\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:36.755933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.755899 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb4e60f-8xx72\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:36.756096 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.755949 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9xkq\" (UniqueName: \"kubernetes.io/projected/d7d50736-194e-42c4-b972-d0407bb2fe6d-kube-api-access-l9xkq\") pod \"maas-api-revoke-sub-3fb4e60f-8xx72\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:36.756096 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.756053 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb4e60f-8xx72\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:36.756750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.756726 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb4e60f-8xx72\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:36.758388 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.758367 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb4e60f-8xx72\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:36.810062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.810001 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9xkq\" (UniqueName: \"kubernetes.io/projected/d7d50736-194e-42c4-b972-d0407bb2fe6d-kube-api-access-l9xkq\") pod \"maas-api-revoke-sub-3fb4e60f-8xx72\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:36.871252 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:36.871216 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:37.245379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.245349 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72"] Nov 02 23:13:37.246789 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:13:37.246744 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d50736_194e_42c4_b972_d0407bb2fe6d.slice/crio-b61ce13ea266fcb087ccde2bdc6178639c66c909d58163f4e3d8f2eec35d5ddf WatchSource:0}: Error finding container b61ce13ea266fcb087ccde2bdc6178639c66c909d58163f4e3d8f2eec35d5ddf: Status 404 returned error can't find the container with id b61ce13ea266fcb087ccde2bdc6178639c66c909d58163f4e3d8f2eec35d5ddf Nov 02 23:13:37.422987 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.422947 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt"] Nov 02 23:13:37.428318 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.428296 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:37.479001 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.478976 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt"] Nov 02 23:13:37.565274 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.565244 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb0b6961-k9bxt\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:37.565565 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.565281 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb0b6961-k9bxt\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:37.565565 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.565308 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkkjv\" (UniqueName: \"kubernetes.io/projected/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-kube-api-access-fkkjv\") pod \"maas-api-revoke-sub-eb0b6961-k9bxt\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:37.666634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.666603 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb0b6961-k9bxt\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:37.666634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.666641 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb0b6961-k9bxt\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:37.666866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.666666 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkkjv\" (UniqueName: \"kubernetes.io/projected/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-kube-api-access-fkkjv\") pod \"maas-api-revoke-sub-eb0b6961-k9bxt\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:37.667393 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.667363 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb0b6961-k9bxt\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:37.668908 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.668888 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb0b6961-k9bxt\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:37.719651 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.719626 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkkjv\" (UniqueName: \"kubernetes.io/projected/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-kube-api-access-fkkjv\") pod \"maas-api-revoke-sub-eb0b6961-k9bxt\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:37.753269 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.753233 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:37.840563 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.840520 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" event={"ID":"d7d50736-194e-42c4-b972-d0407bb2fe6d","Type":"ContainerStarted","Data":"241cfde8d10f5ac95dce50dfd92c077acbe650c4a03c013e6a454a9cadb853ee"} Nov 02 23:13:37.840699 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.840584 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" event={"ID":"d7d50736-194e-42c4-b972-d0407bb2fe6d","Type":"ContainerStarted","Data":"b61ce13ea266fcb087ccde2bdc6178639c66c909d58163f4e3d8f2eec35d5ddf"} Nov 02 23:13:37.949311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:37.949258 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" podStartSLOduration=1.949243831 podStartE2EDuration="1.949243831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:37.945502063 +0000 UTC m=+1211.352813975" watchObservedRunningTime="2026-11-02 23:13:37.949243831 +0000 UTC m=+1211.356555660" Nov 02 23:13:38.119691 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:38.119663 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt"] Nov 02 23:13:38.152288 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:13:38.152256 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a42f33_57a4_4ec3_a2ab_6704ea60100e.slice/crio-05e3f0ef67997a7896be05ccc85c2d4c3718a36bbd89ceb6a7d4bc228ac32a0f WatchSource:0}: Error finding container 05e3f0ef67997a7896be05ccc85c2d4c3718a36bbd89ceb6a7d4bc228ac32a0f: Status 404 returned error can't find the container with id 05e3f0ef67997a7896be05ccc85c2d4c3718a36bbd89ceb6a7d4bc228ac32a0f Nov 02 23:13:38.850354 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:38.850316 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" event={"ID":"a5a42f33-57a4-4ec3-a2ab-6704ea60100e","Type":"ContainerStarted","Data":"4b93176eb042560bb4b3df90c7ce021fe358cff00a0a959031f73aecdfc7701b"} Nov 02 23:13:38.850794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:38.850382 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" event={"ID":"a5a42f33-57a4-4ec3-a2ab-6704ea60100e","Type":"ContainerStarted","Data":"05e3f0ef67997a7896be05ccc85c2d4c3718a36bbd89ceb6a7d4bc228ac32a0f"} Nov 02 23:13:38.852125 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:38.852093 5672 generic.go:356] "Generic (PLEG): container finished" podID="d7d50736-194e-42c4-b972-d0407bb2fe6d" containerID="241cfde8d10f5ac95dce50dfd92c077acbe650c4a03c013e6a454a9cadb853ee" exitCode=0 Nov 02 23:13:38.852270 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:38.852128 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" event={"ID":"d7d50736-194e-42c4-b972-d0407bb2fe6d","Type":"ContainerDied","Data":"241cfde8d10f5ac95dce50dfd92c077acbe650c4a03c013e6a454a9cadb853ee"} Nov 02 23:13:38.957381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:38.957317 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" podStartSLOduration=1.957294171 podStartE2EDuration="1.957294171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:38.95605966 +0000 UTC m=+1212.363371491" watchObservedRunningTime="2026-11-02 23:13:38.957294171 +0000 UTC m=+1212.364606001" Nov 02 23:13:39.862522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:39.862484 5672 generic.go:356] "Generic (PLEG): container finished" podID="a5a42f33-57a4-4ec3-a2ab-6704ea60100e" containerID="4b93176eb042560bb4b3df90c7ce021fe358cff00a0a959031f73aecdfc7701b" exitCode=0 Nov 02 23:13:39.862994 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:39.862559 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" event={"ID":"a5a42f33-57a4-4ec3-a2ab-6704ea60100e","Type":"ContainerDied","Data":"4b93176eb042560bb4b3df90c7ce021fe358cff00a0a959031f73aecdfc7701b"} Nov 02 23:13:40.024577 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.024548 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:40.098300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.098258 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d50736-194e-42c4-b972-d0407bb2fe6d-kube-api-access-l9xkq\" (UniqueName: \"kubernetes.io/projected/d7d50736-194e-42c4-b972-d0407bb2fe6d-kube-api-access-l9xkq\") pod \"d7d50736-194e-42c4-b972-d0407bb2fe6d\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " Nov 02 23:13:40.098493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.098316 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-xks-ca\") pod \"d7d50736-194e-42c4-b972-d0407bb2fe6d\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " Nov 02 23:13:40.098493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.098381 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-openshift-ca\") pod \"d7d50736-194e-42c4-b972-d0407bb2fe6d\" (UID: \"d7d50736-194e-42c4-b972-d0407bb2fe6d\") " Nov 02 23:13:40.098805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.098779 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-openshift-ca" pod "d7d50736-194e-42c4-b972-d0407bb2fe6d" (UID: "d7d50736-194e-42c4-b972-d0407bb2fe6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:13:40.100335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.100304 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d50736-194e-42c4-b972-d0407bb2fe6d-kube-api-access-l9xkq" pod "d7d50736-194e-42c4-b972-d0407bb2fe6d" (UID: "d7d50736-194e-42c4-b972-d0407bb2fe6d"). InnerVolumeSpecName "kube-api-access-l9xkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:13:40.100450 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.100389 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-xks-ca" pod "d7d50736-194e-42c4-b972-d0407bb2fe6d" (UID: "d7d50736-194e-42c4-b972-d0407bb2fe6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:13:40.200537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.200450 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9xkq\" (UniqueName: \"kubernetes.io/projected/d7d50736-194e-42c4-b972-d0407bb2fe6d-kube-api-access-l9xkq\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:40.200537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.200482 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:40.200537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.200493 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d50736-194e-42c4-b972-d0407bb2fe6d-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:40.871417 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.871385 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" Nov 02 23:13:40.871417 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.871410 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72" event={"ID":"d7d50736-194e-42c4-b972-d0407bb2fe6d","Type":"ContainerDied","Data":"b61ce13ea266fcb087ccde2bdc6178639c66c909d58163f4e3d8f2eec35d5ddf"} Nov 02 23:13:40.871909 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:40.871459 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b61ce13ea266fcb087ccde2bdc6178639c66c909d58163f4e3d8f2eec35d5ddf" Nov 02 23:13:41.052639 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.052613 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:41.212202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.212138 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-openshift-ca\") pod \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " Nov 02 23:13:41.212399 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.212213 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-xks-ca\") pod \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " Nov 02 23:13:41.212399 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.212247 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-kube-api-access-fkkjv\" (UniqueName: \"kubernetes.io/projected/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-kube-api-access-fkkjv\") pod \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\" (UID: \"a5a42f33-57a4-4ec3-a2ab-6704ea60100e\") " Nov 02 23:13:41.212555 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.212532 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-openshift-ca" pod "a5a42f33-57a4-4ec3-a2ab-6704ea60100e" (UID: "a5a42f33-57a4-4ec3-a2ab-6704ea60100e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:13:41.212926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.212900 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:41.214393 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.214369 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-xks-ca" pod "a5a42f33-57a4-4ec3-a2ab-6704ea60100e" (UID: "a5a42f33-57a4-4ec3-a2ab-6704ea60100e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:13:41.214527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.214506 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-kube-api-access-fkkjv" pod "a5a42f33-57a4-4ec3-a2ab-6704ea60100e" (UID: "a5a42f33-57a4-4ec3-a2ab-6704ea60100e"). InnerVolumeSpecName "kube-api-access-fkkjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:13:41.314314 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.314229 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:41.314314 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.314257 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkkjv\" (UniqueName: \"kubernetes.io/projected/a5a42f33-57a4-4ec3-a2ab-6704ea60100e-kube-api-access-fkkjv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:41.880840 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.880811 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" Nov 02 23:13:41.880840 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.880823 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt" event={"ID":"a5a42f33-57a4-4ec3-a2ab-6704ea60100e","Type":"ContainerDied","Data":"05e3f0ef67997a7896be05ccc85c2d4c3718a36bbd89ceb6a7d4bc228ac32a0f"} Nov 02 23:13:41.881316 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:41.880867 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05e3f0ef67997a7896be05ccc85c2d4c3718a36bbd89ceb6a7d4bc228ac32a0f" Nov 02 23:13:45.801665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:45.801630 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:13:47.884122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:47.884072 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:48.395173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:48.394241 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:48.643310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:48.643260 5672 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:48.697754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:48.695577 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:48.813027 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:48.812322 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:48.821993 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:48.821666 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:48.823290 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:48.822611 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:48.868688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:48.868642 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:48.891135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:48.889224 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:48.891135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:48.889649 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:13:48.969734 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:48.964435 5672 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:49.042185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.028241 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:49.042185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.035449 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:49.396773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.396658 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:49.398374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.398348 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:13:49.607457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.607411 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6"] Nov 02 23:13:49.608502 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.608467 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a5a42f33-57a4-4ec3-a2ab-6704ea60100e" containerName="revoke-keys" Nov 02 23:13:49.608648 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.608631 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a42f33-57a4-4ec3-a2ab-6704ea60100e" containerName="revoke-keys" Nov 02 23:13:49.608794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.608784 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d7d50736-194e-42c4-b972-d0407bb2fe6d" containerName="revoke-keys" Nov 02 23:13:49.608872 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.608862 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d50736-194e-42c4-b972-d0407bb2fe6d" containerName="revoke-keys" Nov 02 23:13:49.609085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.609073 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d50736-194e-42c4-b972-d0407bb2fe6d" containerName="revoke-keys" Nov 02 23:13:49.609195 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.609184 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a42f33-57a4-4ec3-a2ab-6704ea60100e" containerName="revoke-keys" Nov 02 23:13:49.673716 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.673629 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.696280 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.696231 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 23:13:49.738297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.738240 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:13:49.739803 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.739774 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b2a689-openshift-default-dockercfg-shh7p\"" Nov 02 23:13:49.823292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.823245 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:49.824238 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.824216 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:13:49.825617 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.825592 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 02 23:13:49.826020 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.826001 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:13:49.858997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.858035 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-socket\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.858997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.858100 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-certs\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.858997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.858138 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-credential-socket\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.858997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.858185 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db24bfe0-5f82-45c5-b498-4c8b6310177c-istiod-ca-cert\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.858997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.858233 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-data\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.858997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.858335 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-token\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.858997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.858365 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m26dn\" (UniqueName: \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-kube-api-access-m26dn\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.858997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.858426 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-envoy\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.858997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.858506 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-podinfo\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.879273 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.879227 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:49.928231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.886717 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6"] Nov 02 23:13:49.928833 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.928808 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:13:49.941175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.896025 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:49.996174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.988285 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-token\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.996174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.988343 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m26dn\" (UniqueName: \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-kube-api-access-m26dn\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.996174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.988420 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-envoy\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.996174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.988532 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-podinfo\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.996174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.988784 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-socket\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.996174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.988836 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-certs\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.996174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.988875 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-credential-socket\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.996174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.988900 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db24bfe0-5f82-45c5-b498-4c8b6310177c-istiod-ca-cert\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:49.996174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.988938 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-data\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:50.001182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:49.997128 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-socket\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:50.001677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.001651 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-certs\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:50.002070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.002048 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-credential-socket\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:50.021019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.002840 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db24bfe0-5f82-45c5-b498-4c8b6310177c-istiod-ca-cert\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:50.021294 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.003261 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-data\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:50.024828 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.024767 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-podinfo\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:50.034182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.026820 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-envoy\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:50.042890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.041333 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:50.042890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.041699 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:13:50.042890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.042643 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:50.042890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.042787 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:13:50.143960 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.143844 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-token\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:50.424207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.413280 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 23:13:50.599173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.571475 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m26dn\" (UniqueName: \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-kube-api-access-m26dn\") pod \"e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:50.599173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.578727 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 02 23:13:50.630622 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.630582 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:50.706436 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.706336 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 23:13:50.823740 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.823696 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:50.833475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.833435 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:50.935176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.898949 5672 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:50.938432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.937700 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:50.938720 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:50.937723 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 23:13:51.044531 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:51.044487 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 23:13:51.561179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:51.537850 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 02 23:13:51.624182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:51.617126 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 02 23:13:51.624539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:51.624516 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 02 23:13:51.625187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:51.625170 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 02 23:13:51.626767 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:51.626749 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 02 23:13:51.626948 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:51.626937 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 02 23:13:51.627083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:51.627071 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 02 23:13:51.922214 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:13:51.916401 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb24bfe0_5f82_45c5_b498_4c8b6310177c.slice/crio-efa997097f9654d04c373ddfc41e3e7400b38e6bad890b4294f7b8d2d1a7d4d3 WatchSource:0}: Error finding container efa997097f9654d04c373ddfc41e3e7400b38e6bad890b4294f7b8d2d1a7d4d3: Status 404 returned error can't find the container with id efa997097f9654d04c373ddfc41e3e7400b38e6bad890b4294f7b8d2d1a7d4d3 Nov 02 23:13:51.941035 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:51.940187 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:13:51.941035 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:51.940285 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:13:51.941035 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:51.940331 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:13:52.108926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.108290 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6"] Nov 02 23:13:52.507655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.506881 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z"] Nov 02 23:13:52.516270 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.516240 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:52.516739 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.516709 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z"] Nov 02 23:13:52.557855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.557741 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh9pc\" (UniqueName: \"kubernetes.io/projected/90f9dfd2-af25-42bc-9bce-0945893921e1-kube-api-access-nh9pc\") pod \"maas-api-revoke-sub-b6e4adb9-njl4z\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:52.558370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.558193 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e4adb9-njl4z\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:52.558696 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.558678 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e4adb9-njl4z\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:52.567433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.566869 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:13:52.567433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.566989 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:13:52.567433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.567092 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:13:52.567433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.567185 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:13:52.617824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.616873 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5"] Nov 02 23:13:52.621211 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.621185 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.648494 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.648446 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" event={"ID":"db24bfe0-5f82-45c5-b498-4c8b6310177c","Type":"ContainerStarted","Data":"5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3"} Nov 02 23:13:52.648680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.648517 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" event={"ID":"db24bfe0-5f82-45c5-b498-4c8b6310177c","Type":"ContainerStarted","Data":"efa997097f9654d04c373ddfc41e3e7400b38e6bad890b4294f7b8d2d1a7d4d3"} Nov 02 23:13:52.654695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.654669 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:52.660625 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.660592 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-routing-overlay-volume\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.660970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.660938 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh9pc\" (UniqueName: \"kubernetes.io/projected/90f9dfd2-af25-42bc-9bce-0945893921e1-kube-api-access-nh9pc\") pod \"maas-api-revoke-sub-b6e4adb9-njl4z\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:52.661086 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.661067 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-plugins-config-volume\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.661200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.661110 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e4adb9-njl4z\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:52.661200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.661138 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-provider-credentials\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.661330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.661215 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e4adb9-njl4z\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:52.661330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.661250 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc88v\" (UniqueName: \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-kube-api-access-tc88v\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.664213 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.664186 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e4adb9-njl4z\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:52.680756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.672769 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 23:13:52.680756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.672828 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4e90176b6698\"" Nov 02 23:13:52.680756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.672918 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 23:13:52.680756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.673014 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-d5hk8\"" Nov 02 23:13:52.680756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.673073 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 23:13:52.680756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.673886 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e4adb9-njl4z\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:52.680756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.675240 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 23:13:52.688930 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.687464 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5"] Nov 02 23:13:52.722181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.721792 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh9pc\" (UniqueName: \"kubernetes.io/projected/90f9dfd2-af25-42bc-9bce-0945893921e1-kube-api-access-nh9pc\") pod \"maas-api-revoke-sub-b6e4adb9-njl4z\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:52.765231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.762500 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-routing-overlay-volume\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.765231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.762744 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-plugins-config-volume\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.765231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.762786 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-provider-credentials\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.765231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.762882 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc88v\" (UniqueName: \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-kube-api-access-tc88v\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.765231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.763997 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-routing-overlay-volume\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.779004 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.768358 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-plugins-config-volume\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.783980 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.783909 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-provider-credentials\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.854466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.846405 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc88v\" (UniqueName: \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-kube-api-access-tc88v\") pod \"payload-processing-external-model-6bcb8dc99c-crsl5\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:52.854466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.848040 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" podStartSLOduration=3.848022733 podStartE2EDuration="3.848022733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:52.845626776 +0000 UTC m=+1226.252938613" watchObservedRunningTime="2026-11-02 23:13:52.848022733 +0000 UTC m=+1226.255334563" Nov 02 23:13:52.854466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.848235 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:52.945231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:52.940350 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:13:53.254197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:53.254166 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z"] Nov 02 23:13:53.255409 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:13:53.255377 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f9dfd2_af25_42bc_9bce_0945893921e1.slice/crio-cf31a5c75efaf36d39231278c476a75e661cfd11a14fb2cb2d449e5ba67a7df2 WatchSource:0}: Error finding container cf31a5c75efaf36d39231278c476a75e661cfd11a14fb2cb2d449e5ba67a7df2: Status 404 returned error can't find the container with id cf31a5c75efaf36d39231278c476a75e661cfd11a14fb2cb2d449e5ba67a7df2 Nov 02 23:13:53.339397 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:13:53.339344 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc257520a_5bcd_4e82_8f59_931b16b7472b.slice/crio-550413032cfac8a7be47b81e2c1641060b1b8c34124fa494e31ddaf849e6bf23 WatchSource:0}: Error finding container 550413032cfac8a7be47b81e2c1641060b1b8c34124fa494e31ddaf849e6bf23: Status 404 returned error can't find the container with id 550413032cfac8a7be47b81e2c1641060b1b8c34124fa494e31ddaf849e6bf23 Nov 02 23:13:53.339635 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:53.339608 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5"] Nov 02 23:13:53.341755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:53.341736 5672 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:13:53.657143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:53.657105 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" event={"ID":"c257520a-5bcd-4e82-8f59-931b16b7472b","Type":"ContainerStarted","Data":"550413032cfac8a7be47b81e2c1641060b1b8c34124fa494e31ddaf849e6bf23"} Nov 02 23:13:53.658933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:53.658897 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" event={"ID":"90f9dfd2-af25-42bc-9bce-0945893921e1","Type":"ContainerStarted","Data":"e3f0553f6403b9e6c7e3a16b50796594f52216f03bd7f27f51c18fa210be9ff8"} Nov 02 23:13:53.659098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:53.658948 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" event={"ID":"90f9dfd2-af25-42bc-9bce-0945893921e1","Type":"ContainerStarted","Data":"cf31a5c75efaf36d39231278c476a75e661cfd11a14fb2cb2d449e5ba67a7df2"} Nov 02 23:13:53.661088 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:53.661067 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:53.772093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:53.772042 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" podStartSLOduration=1.772029208 podStartE2EDuration="1.772029208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:53.768895527 +0000 UTC m=+1227.176207358" watchObservedRunningTime="2026-11-02 23:13:53.772029208 +0000 UTC m=+1227.179341038" Nov 02 23:13:54.680441 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:54.680401 5672 generic.go:356] "Generic (PLEG): container finished" podID="90f9dfd2-af25-42bc-9bce-0945893921e1" containerID="e3f0553f6403b9e6c7e3a16b50796594f52216f03bd7f27f51c18fa210be9ff8" exitCode=0 Nov 02 23:13:54.680921 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:54.680485 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" event={"ID":"90f9dfd2-af25-42bc-9bce-0945893921e1","Type":"ContainerDied","Data":"e3f0553f6403b9e6c7e3a16b50796594f52216f03bd7f27f51c18fa210be9ff8"} Nov 02 23:13:54.681297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:54.681276 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:54.682216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:54.682184 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:13:55.885888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:55.885857 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:55.912143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:55.912110 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-xks-ca\") pod \"90f9dfd2-af25-42bc-9bce-0945893921e1\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " Nov 02 23:13:55.912343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:55.912203 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f9dfd2-af25-42bc-9bce-0945893921e1-kube-api-access-nh9pc\" (UniqueName: \"kubernetes.io/projected/90f9dfd2-af25-42bc-9bce-0945893921e1-kube-api-access-nh9pc\") pod \"90f9dfd2-af25-42bc-9bce-0945893921e1\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " Nov 02 23:13:55.912450 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:55.912427 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-openshift-ca\") pod \"90f9dfd2-af25-42bc-9bce-0945893921e1\" (UID: \"90f9dfd2-af25-42bc-9bce-0945893921e1\") " Nov 02 23:13:55.912945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:55.912918 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-openshift-ca" pod "90f9dfd2-af25-42bc-9bce-0945893921e1" (UID: "90f9dfd2-af25-42bc-9bce-0945893921e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:13:55.914786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:55.914758 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-xks-ca" pod "90f9dfd2-af25-42bc-9bce-0945893921e1" (UID: "90f9dfd2-af25-42bc-9bce-0945893921e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:13:55.915558 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:55.915531 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f9dfd2-af25-42bc-9bce-0945893921e1-kube-api-access-nh9pc" pod "90f9dfd2-af25-42bc-9bce-0945893921e1" (UID: "90f9dfd2-af25-42bc-9bce-0945893921e1"). InnerVolumeSpecName "kube-api-access-nh9pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:13:56.029170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:56.017394 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:56.029170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:56.017429 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f9dfd2-af25-42bc-9bce-0945893921e1-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:56.029170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:56.017444 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh9pc\" (UniqueName: \"kubernetes.io/projected/90f9dfd2-af25-42bc-9bce-0945893921e1-kube-api-access-nh9pc\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:13:56.181404 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:13:56.181339 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:13:56.699461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:56.699430 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" Nov 02 23:13:56.699596 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:56.699481 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z" event={"ID":"90f9dfd2-af25-42bc-9bce-0945893921e1","Type":"ContainerDied","Data":"cf31a5c75efaf36d39231278c476a75e661cfd11a14fb2cb2d449e5ba67a7df2"} Nov 02 23:13:56.699596 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:56.699526 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf31a5c75efaf36d39231278c476a75e661cfd11a14fb2cb2d449e5ba67a7df2" Nov 02 23:13:56.701516 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:56.701492 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" event={"ID":"c257520a-5bcd-4e82-8f59-931b16b7472b","Type":"ContainerStarted","Data":"e2d762b1b27cb48d82c844c46d260870b7fd2177f16fd7df2769bae4b1f6021f"} Nov 02 23:13:56.814040 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:56.813987 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" podStartSLOduration=1.548052034 podStartE2EDuration="4.813973859s" totalImagesPullingTime="3.265921825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:56.812301385 +0000 UTC m=+1230.219613235" watchObservedRunningTime="2026-11-02 23:13:56.813973859 +0000 UTC m=+1230.221285721" Nov 02 23:13:57.708762 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:13:57.708733 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:14:02.602630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.602595 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl"] Nov 02 23:14:02.603475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.603452 5672 cpu_manager.go:414] "containerMap: removing container" podUID="90f9dfd2-af25-42bc-9bce-0945893921e1" containerName="revoke-keys" Nov 02 23:14:02.603571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.603477 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f9dfd2-af25-42bc-9bce-0945893921e1" containerName="revoke-keys" Nov 02 23:14:02.603645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.603631 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f9dfd2-af25-42bc-9bce-0945893921e1" containerName="revoke-keys" Nov 02 23:14:02.608160 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.608127 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:14:02.660541 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.660511 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl"] Nov 02 23:14:02.701108 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.701045 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b2a689\"" Nov 02 23:14:02.701384 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.701183 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b2a689-dockercfg-d5462\"" Nov 02 23:14:02.716747 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.716714 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275c4d8b-85fb-48fa-b065-22f798a57ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl\" (UID: \"275c4d8b-85fb-48fa-b065-22f798a57ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:14:02.716877 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.716823 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58jg\" (UniqueName: \"kubernetes.io/projected/275c4d8b-85fb-48fa-b065-22f798a57ebb-kube-api-access-f58jg\") pod \"payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl\" (UID: \"275c4d8b-85fb-48fa-b065-22f798a57ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:14:02.766079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.766051 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl"] Nov 02 23:14:02.770313 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.770280 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:14:02.819111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.819075 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58jg\" (UniqueName: \"kubernetes.io/projected/275c4d8b-85fb-48fa-b065-22f798a57ebb-kube-api-access-f58jg\") pod \"payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl\" (UID: \"275c4d8b-85fb-48fa-b065-22f798a57ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:14:02.819280 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.819169 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl\" (UID: \"ed428b35-b9e9-4012-ac33-b9dd5b702d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:14:02.819280 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.819242 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbcb\" (UniqueName: \"kubernetes.io/projected/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-kube-api-access-ztbcb\") pod \"payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl\" (UID: \"ed428b35-b9e9-4012-ac33-b9dd5b702d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:14:02.819789 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.819762 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275c4d8b-85fb-48fa-b065-22f798a57ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl\" (UID: \"275c4d8b-85fb-48fa-b065-22f798a57ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:14:02.820552 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.820525 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275c4d8b-85fb-48fa-b065-22f798a57ebb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl\" (UID: \"275c4d8b-85fb-48fa-b065-22f798a57ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:14:02.823502 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.823480 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl"] Nov 02 23:14:02.874404 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.874318 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58jg\" (UniqueName: \"kubernetes.io/projected/275c4d8b-85fb-48fa-b065-22f798a57ebb-kube-api-access-f58jg\") pod \"payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl\" (UID: \"275c4d8b-85fb-48fa-b065-22f798a57ebb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:14:02.921247 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.921210 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl\" (UID: \"ed428b35-b9e9-4012-ac33-b9dd5b702d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:14:02.921453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.921279 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbcb\" (UniqueName: \"kubernetes.io/projected/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-kube-api-access-ztbcb\") pod \"payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl\" (UID: \"ed428b35-b9e9-4012-ac33-b9dd5b702d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:14:02.922677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.922649 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl\" (UID: \"ed428b35-b9e9-4012-ac33-b9dd5b702d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:14:02.923792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.923770 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:14:02.980070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:02.980001 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbcb\" (UniqueName: \"kubernetes.io/projected/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-kube-api-access-ztbcb\") pod \"payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl\" (UID: \"ed428b35-b9e9-4012-ac33-b9dd5b702d6f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:14:03.084168 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:03.084118 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:14:03.327695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:03.327587 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl"] Nov 02 23:14:03.327847 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:14:03.327775 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275c4d8b_85fb_48fa_b065_22f798a57ebb.slice/crio-af48c9cd7af00422c9f9dd7eba4fdfc50c748661c5fe5abf4a66ecb4d03b23de WatchSource:0}: Error finding container af48c9cd7af00422c9f9dd7eba4fdfc50c748661c5fe5abf4a66ecb4d03b23de: Status 404 returned error can't find the container with id af48c9cd7af00422c9f9dd7eba4fdfc50c748661c5fe5abf4a66ecb4d03b23de Nov 02 23:14:03.454102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:03.454071 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl"] Nov 02 23:14:03.454478 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:14:03.454451 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded428b35_b9e9_4012_ac33_b9dd5b702d6f.slice/crio-69a05faf01d9a033e25f4df7c4f673161002f629f5c44ff9c563ad87278b3db2 WatchSource:0}: Error finding container 69a05faf01d9a033e25f4df7c4f673161002f629f5c44ff9c563ad87278b3db2: Status 404 returned error can't find the container with id 69a05faf01d9a033e25f4df7c4f673161002f629f5c44ff9c563ad87278b3db2 Nov 02 23:14:03.755708 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:03.755667 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" event={"ID":"ed428b35-b9e9-4012-ac33-b9dd5b702d6f","Type":"ContainerStarted","Data":"34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786"} Nov 02 23:14:03.756189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:03.755725 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:14:03.756189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:03.755751 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" event={"ID":"ed428b35-b9e9-4012-ac33-b9dd5b702d6f","Type":"ContainerStarted","Data":"69a05faf01d9a033e25f4df7c4f673161002f629f5c44ff9c563ad87278b3db2"} Nov 02 23:14:03.757286 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:03.757262 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" event={"ID":"275c4d8b-85fb-48fa-b065-22f798a57ebb","Type":"ContainerStarted","Data":"d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee"} Nov 02 23:14:03.757375 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:03.757301 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" event={"ID":"275c4d8b-85fb-48fa-b065-22f798a57ebb","Type":"ContainerStarted","Data":"af48c9cd7af00422c9f9dd7eba4fdfc50c748661c5fe5abf4a66ecb4d03b23de"} Nov 02 23:14:03.757417 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:03.757380 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:14:03.867116 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:03.867065 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" podStartSLOduration=1.86705124 podStartE2EDuration="1.86705124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:03.864258441 +0000 UTC m=+1237.271570283" watchObservedRunningTime="2026-11-02 23:14:03.86705124 +0000 UTC m=+1237.274363081" Nov 02 23:14:03.974871 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:03.974819 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" podStartSLOduration=1.974805313 podStartE2EDuration="1.974805313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:03.972619259 +0000 UTC m=+1237.379931094" watchObservedRunningTime="2026-11-02 23:14:03.974805313 +0000 UTC m=+1237.382117143" Nov 02 23:14:05.579816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:05.579775 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5"] Nov 02 23:14:05.583075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:05.583044 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" podUID="c257520a-5bcd-4e82-8f59-931b16b7472b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:14:10.580723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:10.580676 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" podUID="c257520a-5bcd-4e82-8f59-931b16b7472b" containerName="payload-processing" containerID="cri-o://e2d762b1b27cb48d82c844c46d260870b7fd2177f16fd7df2769bae4b1f6021f" gracePeriod=25 Nov 02 23:14:10.817006 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:10.816969 5672 generic.go:356] "Generic (PLEG): container finished" podID="c257520a-5bcd-4e82-8f59-931b16b7472b" containerID="e2d762b1b27cb48d82c844c46d260870b7fd2177f16fd7df2769bae4b1f6021f" exitCode=0 Nov 02 23:14:10.817233 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:10.817212 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" event={"ID":"c257520a-5bcd-4e82-8f59-931b16b7472b","Type":"ContainerDied","Data":"e2d762b1b27cb48d82c844c46d260870b7fd2177f16fd7df2769bae4b1f6021f"} Nov 02 23:14:10.851124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:10.851099 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:14:11.036472 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.036442 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-kube-api-access-tc88v\" (UniqueName: \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-kube-api-access-tc88v\") pod \"c257520a-5bcd-4e82-8f59-931b16b7472b\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " Nov 02 23:14:11.036636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.036505 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-plugins-config-volume\") pod \"c257520a-5bcd-4e82-8f59-931b16b7472b\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " Nov 02 23:14:11.036636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.036539 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-provider-credentials\") pod \"c257520a-5bcd-4e82-8f59-931b16b7472b\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " Nov 02 23:14:11.036708 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.036692 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-routing-overlay-volume\") pod \"c257520a-5bcd-4e82-8f59-931b16b7472b\" (UID: \"c257520a-5bcd-4e82-8f59-931b16b7472b\") " Nov 02 23:14:11.037019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.036986 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-plugins-config-volume" pod "c257520a-5bcd-4e82-8f59-931b16b7472b" (UID: "c257520a-5bcd-4e82-8f59-931b16b7472b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:14:11.037180 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.037038 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-routing-overlay-volume" pod "c257520a-5bcd-4e82-8f59-931b16b7472b" (UID: "c257520a-5bcd-4e82-8f59-931b16b7472b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:14:11.037289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.037270 5672 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-routing-overlay-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:11.037350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.037289 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c257520a-5bcd-4e82-8f59-931b16b7472b-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:11.038784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.038731 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-provider-credentials" pod "c257520a-5bcd-4e82-8f59-931b16b7472b" (UID: "c257520a-5bcd-4e82-8f59-931b16b7472b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:14:11.038872 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.038845 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-kube-api-access-tc88v" pod "c257520a-5bcd-4e82-8f59-931b16b7472b" (UID: "c257520a-5bcd-4e82-8f59-931b16b7472b"). InnerVolumeSpecName "kube-api-access-tc88v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:14:11.138943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.138879 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc88v\" (UniqueName: \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-kube-api-access-tc88v\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:11.138943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.138906 5672 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c257520a-5bcd-4e82-8f59-931b16b7472b-provider-credentials\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:11.826080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.826044 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" event={"ID":"c257520a-5bcd-4e82-8f59-931b16b7472b","Type":"ContainerDied","Data":"550413032cfac8a7be47b81e2c1641060b1b8c34124fa494e31ddaf849e6bf23"} Nov 02 23:14:11.826080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.826064 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5" Nov 02 23:14:11.826620 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.826106 5672 scope.go:122] "RemoveContainer" containerID="e2d762b1b27cb48d82c844c46d260870b7fd2177f16fd7df2769bae4b1f6021f" Nov 02 23:14:11.981103 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:11.981071 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5"] Nov 02 23:14:12.027163 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:12.027123 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-crsl5"] Nov 02 23:14:12.713132 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:12.713097 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c257520a-5bcd-4e82-8f59-931b16b7472b" path="/var/lib/kubelet/pods/c257520a-5bcd-4e82-8f59-931b16b7472b/volumes" Nov 02 23:14:14.767443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:14.767416 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:14:14.767849 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:14.767490 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:14:26.020791 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:14:26.020749 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:14:27.145209 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:27.145173 5672 scope.go:122] "RemoveContainer" containerID="af4c668dc70796fd542b75564d145f31ecf68e4cf5e7b98e0bcc7d2dc4c96a88" Nov 02 23:14:27.155741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:27.155719 5672 scope.go:122] "RemoveContainer" containerID="2afcfd3195d5a8998996d3d054155224abd859802c8f535cf726f9f7c70b1d97" Nov 02 23:14:27.165402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:27.165382 5672 scope.go:122] "RemoveContainer" containerID="f834794891e434edc85cf16a98ef52b792c2364155fd943e3282f3f3519331d4" Nov 02 23:14:27.177396 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:27.177284 5672 scope.go:122] "RemoveContainer" containerID="8209a8616fa18ea1b0978f32270c00ccb7aa36eb2c4b5138c088fb6fd82a9779" Nov 02 23:14:41.148471 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.148437 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm"] Nov 02 23:14:41.148993 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.148980 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c257520a-5bcd-4e82-8f59-931b16b7472b" containerName="payload-processing" Nov 02 23:14:41.149066 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.148994 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c257520a-5bcd-4e82-8f59-931b16b7472b" containerName="payload-processing" Nov 02 23:14:41.149137 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.149123 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c257520a-5bcd-4e82-8f59-931b16b7472b" containerName="payload-processing" Nov 02 23:14:41.155323 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.155295 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.173798 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.173764 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm"] Nov 02 23:14:41.204261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.204234 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"distinct-w3-14b321-kserve-self-signed-certs\"" Nov 02 23:14:41.223838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.223810 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:14:41.269115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.269064 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-model-cache\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.269646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.269618 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4fd478e-361d-479a-9703-bb9a72ef8c17-tls-certs\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.269890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.269870 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-dshm\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.270094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.270076 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5hv\" (UniqueName: \"kubernetes.io/projected/a4fd478e-361d-479a-9703-bb9a72ef8c17-kube-api-access-gk5hv\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.270379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.270361 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-tmp-dir\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.270699 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.270675 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-home\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.372701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.372659 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4fd478e-361d-479a-9703-bb9a72ef8c17-tls-certs\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.372917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.372723 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-dshm\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.372917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.372769 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5hv\" (UniqueName: \"kubernetes.io/projected/a4fd478e-361d-479a-9703-bb9a72ef8c17-kube-api-access-gk5hv\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.372917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.372820 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-tmp-dir\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.372917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.372916 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-home\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.373189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.373027 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-model-cache\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.373507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.373486 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-model-cache\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.373750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.373729 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-tmp-dir\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.373944 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.373872 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-home\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.376219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.376193 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-dshm\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.376693 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.376660 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4fd478e-361d-479a-9703-bb9a72ef8c17-tls-certs\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.431254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.427116 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5hv\" (UniqueName: \"kubernetes.io/projected/a4fd478e-361d-479a-9703-bb9a72ef8c17-kube-api-access-gk5hv\") pod \"distinct-w3-14b321-kserve-9b86f6dfc-2ltgm\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.475173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.469407 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:41.928568 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:41.928503 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm"] Nov 02 23:14:42.161800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:42.161766 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" event={"ID":"a4fd478e-361d-479a-9703-bb9a72ef8c17","Type":"ContainerStarted","Data":"87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a"} Nov 02 23:14:42.162239 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:42.161815 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" event={"ID":"a4fd478e-361d-479a-9703-bb9a72ef8c17","Type":"ContainerStarted","Data":"9e7bf34f344f0b5c641e99b87cd70543f102822f8e2ca07c79871052bab838ec"} Nov 02 23:14:42.270927 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:42.270856 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" podStartSLOduration=1.27083524 podStartE2EDuration="1.27083524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:42.268486482 +0000 UTC m=+1275.675798312" watchObservedRunningTime="2026-11-02 23:14:42.27083524 +0000 UTC m=+1275.678147073" Nov 02 23:14:42.470554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:42.470519 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:49.936758 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:49.936714 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l"] Nov 02 23:14:49.941317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:49.941282 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:49.990808 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:49.990475 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:14:49.997009 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:49.996970 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l"] Nov 02 23:14:50.033232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.033186 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj8cs\" (UniqueName: \"kubernetes.io/projected/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-kube-api-access-fj8cs\") pod \"maas-api-revoke-sub-80e60c3c-lnp5l\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:50.033403 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.033376 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80e60c3c-lnp5l\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:50.033500 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.033485 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80e60c3c-lnp5l\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:50.037073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.037044 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:14:50.037073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.037071 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:14:50.037073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.037084 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:14:50.135191 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.135143 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80e60c3c-lnp5l\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:50.135353 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.135311 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj8cs\" (UniqueName: \"kubernetes.io/projected/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-kube-api-access-fj8cs\") pod \"maas-api-revoke-sub-80e60c3c-lnp5l\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:50.135442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.135425 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80e60c3c-lnp5l\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:50.135826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.135802 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80e60c3c-lnp5l\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:50.137707 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.137686 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80e60c3c-lnp5l\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:50.191087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.191015 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj8cs\" (UniqueName: \"kubernetes.io/projected/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-kube-api-access-fj8cs\") pod \"maas-api-revoke-sub-80e60c3c-lnp5l\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:50.256291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.256257 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:50.627685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:50.627643 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l"] Nov 02 23:14:50.627922 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:14:50.627887 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode738fd68_52f3_4f57_b0a1_a4ab12c091ec.slice/crio-d1c5b6bff7aa31385b5537578afd58473f99567f74b94a01ece5c06db1297ba5 WatchSource:0}: Error finding container d1c5b6bff7aa31385b5537578afd58473f99567f74b94a01ece5c06db1297ba5: Status 404 returned error can't find the container with id d1c5b6bff7aa31385b5537578afd58473f99567f74b94a01ece5c06db1297ba5 Nov 02 23:14:51.232544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:51.232503 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" event={"ID":"e738fd68-52f3-4f57-b0a1-a4ab12c091ec","Type":"ContainerStarted","Data":"1b7286163369779e72cc6b285191dbeba6da78906531a3c4ed9cbdda4b8f352a"} Nov 02 23:14:51.233034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:51.232564 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" event={"ID":"e738fd68-52f3-4f57-b0a1-a4ab12c091ec","Type":"ContainerStarted","Data":"d1c5b6bff7aa31385b5537578afd58473f99567f74b94a01ece5c06db1297ba5"} Nov 02 23:14:51.234204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:51.234173 5672 generic.go:356] "Generic (PLEG): container finished" podID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerID="5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3" exitCode=0 Nov 02 23:14:51.234318 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:51.234218 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" event={"ID":"db24bfe0-5f82-45c5-b498-4c8b6310177c","Type":"ContainerDied","Data":"5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3"} Nov 02 23:14:51.234580 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:51.234567 5672 scope.go:122] "RemoveContainer" containerID="5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3" Nov 02 23:14:51.235885 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:51.235856 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:14:51.235977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:51.235921 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:14:51.235977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:51.235951 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:14:51.338966 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:51.338792 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" podStartSLOduration=2.338778125 podStartE2EDuration="2.338778125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:14:51.337732251 +0000 UTC m=+1284.745044086" watchObservedRunningTime="2026-11-02 23:14:51.338778125 +0000 UTC m=+1284.746089956" Nov 02 23:14:51.470503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:51.470461 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:51.490182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:51.490123 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:52.257056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:52.257022 5672 generic.go:356] "Generic (PLEG): container finished" podID="e738fd68-52f3-4f57-b0a1-a4ab12c091ec" containerID="1b7286163369779e72cc6b285191dbeba6da78906531a3c4ed9cbdda4b8f352a" exitCode=0 Nov 02 23:14:52.257601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:52.257066 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" event={"ID":"e738fd68-52f3-4f57-b0a1-a4ab12c091ec","Type":"ContainerDied","Data":"1b7286163369779e72cc6b285191dbeba6da78906531a3c4ed9cbdda4b8f352a"} Nov 02 23:14:52.259207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:52.259181 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" event={"ID":"db24bfe0-5f82-45c5-b498-4c8b6310177c","Type":"ContainerStarted","Data":"9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b"} Nov 02 23:14:52.270378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:52.270348 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 02 23:14:52.657255 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:52.657168 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:14:52.657423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:52.657291 5672 patch_prober.go:29] interesting pod/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:14:52.657423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:52.657333 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 23:14:53.484334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.484310 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:53.581367 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.581281 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-xks-ca\") pod \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " Nov 02 23:14:53.581547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.581455 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-kube-api-access-fj8cs\" (UniqueName: \"kubernetes.io/projected/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-kube-api-access-fj8cs\") pod \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " Nov 02 23:14:53.581614 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.581559 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-openshift-ca\") pod \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\" (UID: \"e738fd68-52f3-4f57-b0a1-a4ab12c091ec\") " Nov 02 23:14:53.582182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.582139 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-openshift-ca" pod "e738fd68-52f3-4f57-b0a1-a4ab12c091ec" (UID: "e738fd68-52f3-4f57-b0a1-a4ab12c091ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:14:53.585074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.585047 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-xks-ca" pod "e738fd68-52f3-4f57-b0a1-a4ab12c091ec" (UID: "e738fd68-52f3-4f57-b0a1-a4ab12c091ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:14:53.585611 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.585591 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-kube-api-access-fj8cs" pod "e738fd68-52f3-4f57-b0a1-a4ab12c091ec" (UID: "e738fd68-52f3-4f57-b0a1-a4ab12c091ec"). InnerVolumeSpecName "kube-api-access-fj8cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:14:53.654688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.654646 5672 patch_prober.go:29] interesting pod/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:14:53.654861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.654718 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 23:14:53.683640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.683603 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:53.683640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.683633 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj8cs\" (UniqueName: \"kubernetes.io/projected/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-kube-api-access-fj8cs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:53.683640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:53.683644 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e738fd68-52f3-4f57-b0a1-a4ab12c091ec-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:14:54.277375 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:54.277345 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" Nov 02 23:14:54.277542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:54.277373 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l" event={"ID":"e738fd68-52f3-4f57-b0a1-a4ab12c091ec","Type":"ContainerDied","Data":"d1c5b6bff7aa31385b5537578afd58473f99567f74b94a01ece5c06db1297ba5"} Nov 02 23:14:54.277542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:54.277423 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1c5b6bff7aa31385b5537578afd58473f99567f74b94a01ece5c06db1297ba5" Nov 02 23:14:54.655235 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:54.654860 5672 patch_prober.go:29] interesting pod/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:14:54.655235 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:54.654949 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 23:14:54.681551 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:54.681515 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:14:55.655172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:55.654827 5672 patch_prober.go:29] interesting pod/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:14:55.655172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:55.654911 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 23:14:56.058206 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:14:56.047418 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:14:56.655469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:56.655431 5672 patch_prober.go:29] interesting pod/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 02 23:14:56.655868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:56.655491 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 02 23:14:57.659205 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:57.659176 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:14:57.660295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:14:57.660273 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:15:00.175329 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.175288 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g"] Nov 02 23:15:00.175883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.175865 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e738fd68-52f3-4f57-b0a1-a4ab12c091ec" containerName="revoke-keys" Nov 02 23:15:00.175883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.175884 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e738fd68-52f3-4f57-b0a1-a4ab12c091ec" containerName="revoke-keys" Nov 02 23:15:00.176002 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.175985 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e738fd68-52f3-4f57-b0a1-a4ab12c091ec" containerName="revoke-keys" Nov 02 23:15:00.178831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.178809 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" Nov 02 23:15:00.180191 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.180163 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl"] Nov 02 23:15:00.183129 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.183105 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" Nov 02 23:15:00.184103 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.184082 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz"] Nov 02 23:15:00.187202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.187185 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" Nov 02 23:15:00.221493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.221462 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q"] Nov 02 23:15:00.233063 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.233012 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87"] Nov 02 23:15:00.235042 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.235005 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:15:00.235328 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.235304 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" Nov 02 23:15:00.235434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.235317 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:15:00.235630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.235614 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:15:00.241771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.241745 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" Nov 02 23:15:00.269176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.269129 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g"] Nov 02 23:15:00.274206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.270661 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0632195a-41cb-4195-98a7-feb73e326acf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl\" (UID: \"0632195a-41cb-4195-98a7-feb73e326acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" Nov 02 23:15:00.274206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.270799 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q\" (UID: \"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" Nov 02 23:15:00.274206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.270835 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9mtq\" (UniqueName: \"kubernetes.io/projected/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-kube-api-access-p9mtq\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q\" (UID: \"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" Nov 02 23:15:00.274206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.270892 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e91272-bc78-41ad-b139-b28c0f518756-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz\" (UID: \"97e91272-bc78-41ad-b139-b28c0f518756\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" Nov 02 23:15:00.274206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.271117 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbbsq\" (UniqueName: \"kubernetes.io/projected/97e91272-bc78-41ad-b139-b28c0f518756-kube-api-access-xbbsq\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz\" (UID: \"97e91272-bc78-41ad-b139-b28c0f518756\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" Nov 02 23:15:00.274206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.271203 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894355-27r8g\" (UID: \"28f50d8f-82bb-4fcc-ac57-6085d5916f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" Nov 02 23:15:00.274206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.271238 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd7lb\" (UniqueName: \"kubernetes.io/projected/0632195a-41cb-4195-98a7-feb73e326acf-kube-api-access-hd7lb\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl\" (UID: \"0632195a-41cb-4195-98a7-feb73e326acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" Nov 02 23:15:00.274206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.271276 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8np87\" (UniqueName: \"kubernetes.io/projected/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-kube-api-access-8np87\") pod \"maas-api-key-cleanup-29894355-27r8g\" (UID: \"28f50d8f-82bb-4fcc-ac57-6085d5916f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" Nov 02 23:15:00.274206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.274072 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl"] Nov 02 23:15:00.304030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.303947 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz"] Nov 02 23:15:00.314043 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.312212 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87"] Nov 02 23:15:00.341707 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.341676 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q"] Nov 02 23:15:00.374585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.374402 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0632195a-41cb-4195-98a7-feb73e326acf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl\" (UID: \"0632195a-41cb-4195-98a7-feb73e326acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" Nov 02 23:15:00.374585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.374514 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q\" (UID: \"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" Nov 02 23:15:00.374585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.374549 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9mtq\" (UniqueName: \"kubernetes.io/projected/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-kube-api-access-p9mtq\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q\" (UID: \"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" Nov 02 23:15:00.374585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.374586 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87\" (UID: \"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" Nov 02 23:15:00.374968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.374643 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e91272-bc78-41ad-b139-b28c0f518756-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz\" (UID: \"97e91272-bc78-41ad-b139-b28c0f518756\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" Nov 02 23:15:00.374968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.374878 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbbsq\" (UniqueName: \"kubernetes.io/projected/97e91272-bc78-41ad-b139-b28c0f518756-kube-api-access-xbbsq\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz\" (UID: \"97e91272-bc78-41ad-b139-b28c0f518756\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" Nov 02 23:15:00.374968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.374935 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894355-27r8g\" (UID: \"28f50d8f-82bb-4fcc-ac57-6085d5916f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" Nov 02 23:15:00.375136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.374968 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd7lb\" (UniqueName: \"kubernetes.io/projected/0632195a-41cb-4195-98a7-feb73e326acf-kube-api-access-hd7lb\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl\" (UID: \"0632195a-41cb-4195-98a7-feb73e326acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" Nov 02 23:15:00.375136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.375006 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8np87\" (UniqueName: \"kubernetes.io/projected/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-kube-api-access-8np87\") pod \"maas-api-key-cleanup-29894355-27r8g\" (UID: \"28f50d8f-82bb-4fcc-ac57-6085d5916f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" Nov 02 23:15:00.375136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.375044 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ccb\" (UniqueName: \"kubernetes.io/projected/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-kube-api-access-s7ccb\") pod \"maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87\" (UID: \"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" Nov 02 23:15:00.376318 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.375324 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0632195a-41cb-4195-98a7-feb73e326acf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl\" (UID: \"0632195a-41cb-4195-98a7-feb73e326acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" Nov 02 23:15:00.376434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.375883 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q\" (UID: \"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" Nov 02 23:15:00.376434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.376071 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894355-27r8g\" (UID: \"28f50d8f-82bb-4fcc-ac57-6085d5916f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" Nov 02 23:15:00.376718 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.376701 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e91272-bc78-41ad-b139-b28c0f518756-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz\" (UID: \"97e91272-bc78-41ad-b139-b28c0f518756\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" Nov 02 23:15:00.430253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.430179 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9mtq\" (UniqueName: \"kubernetes.io/projected/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-kube-api-access-p9mtq\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q\" (UID: \"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" Nov 02 23:15:00.430680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.430663 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbbsq\" (UniqueName: \"kubernetes.io/projected/97e91272-bc78-41ad-b139-b28c0f518756-kube-api-access-xbbsq\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz\" (UID: \"97e91272-bc78-41ad-b139-b28c0f518756\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" Nov 02 23:15:00.431044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.431020 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8np87\" (UniqueName: \"kubernetes.io/projected/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-kube-api-access-8np87\") pod \"maas-api-key-cleanup-29894355-27r8g\" (UID: \"28f50d8f-82bb-4fcc-ac57-6085d5916f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" Nov 02 23:15:00.432475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.432457 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd7lb\" (UniqueName: \"kubernetes.io/projected/0632195a-41cb-4195-98a7-feb73e326acf-kube-api-access-hd7lb\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl\" (UID: \"0632195a-41cb-4195-98a7-feb73e326acf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" Nov 02 23:15:00.476578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.476541 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ccb\" (UniqueName: \"kubernetes.io/projected/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-kube-api-access-s7ccb\") pod \"maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87\" (UID: \"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" Nov 02 23:15:00.476762 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.476626 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87\" (UID: \"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" Nov 02 23:15:00.477333 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.477307 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87\" (UID: \"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" Nov 02 23:15:00.493828 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.493800 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" Nov 02 23:15:00.500928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.500908 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" Nov 02 23:15:00.509351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.509245 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" Nov 02 23:15:00.532444 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.532414 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ccb\" (UniqueName: \"kubernetes.io/projected/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-kube-api-access-s7ccb\") pod \"maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87\" (UID: \"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" Nov 02 23:15:00.562170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.557824 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" Nov 02 23:15:00.576351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.576307 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" Nov 02 23:15:00.899690 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.899642 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl"] Nov 02 23:15:00.899908 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:15:00.899873 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0632195a_41cb_4195_98a7_feb73e326acf.slice/crio-cee0b90007b1d821fb664ff6784d65f20f9e1203a72b6c68504f2ee5d615461e WatchSource:0}: Error finding container cee0b90007b1d821fb664ff6784d65f20f9e1203a72b6c68504f2ee5d615461e: Status 404 returned error can't find the container with id cee0b90007b1d821fb664ff6784d65f20f9e1203a72b6c68504f2ee5d615461e Nov 02 23:15:00.960592 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.959093 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz"] Nov 02 23:15:00.962683 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:15:00.962649 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f50d8f_82bb_4fcc_ac57_6085d5916f5e.slice/crio-82be05c9d2d392a77a78d605effe936f2c1e22437755bbbc5b1f0b5dab1c4cdd WatchSource:0}: Error finding container 82be05c9d2d392a77a78d605effe936f2c1e22437755bbbc5b1f0b5dab1c4cdd: Status 404 returned error can't find the container with id 82be05c9d2d392a77a78d605effe936f2c1e22437755bbbc5b1f0b5dab1c4cdd Nov 02 23:15:00.968507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:00.968479 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g"] Nov 02 23:15:01.007897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:01.007867 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q"] Nov 02 23:15:01.008896 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:15:01.008854 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502911d5_6f2b_4dd6_8f5b_8fe9c9b50693.slice/crio-0d378ed94a529b75186a5151b8069174c9fd926bafe239d5630ff46c187b90dd WatchSource:0}: Error finding container 0d378ed94a529b75186a5151b8069174c9fd926bafe239d5630ff46c187b90dd: Status 404 returned error can't find the container with id 0d378ed94a529b75186a5151b8069174c9fd926bafe239d5630ff46c187b90dd Nov 02 23:15:01.042634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:01.042603 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87"] Nov 02 23:15:01.336997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:01.336958 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" event={"ID":"28f50d8f-82bb-4fcc-ac57-6085d5916f5e","Type":"ContainerStarted","Data":"82be05c9d2d392a77a78d605effe936f2c1e22437755bbbc5b1f0b5dab1c4cdd"} Nov 02 23:15:01.338704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:01.338675 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" event={"ID":"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693","Type":"ContainerStarted","Data":"0d378ed94a529b75186a5151b8069174c9fd926bafe239d5630ff46c187b90dd"} Nov 02 23:15:01.340171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:01.340133 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" event={"ID":"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575","Type":"ContainerStarted","Data":"65086afc20c1e2fa86f80eb0be88e8a155d8dce087ebaa7073ecd6c3d18ef885"} Nov 02 23:15:01.341893 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:01.341870 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" event={"ID":"0632195a-41cb-4195-98a7-feb73e326acf","Type":"ContainerStarted","Data":"cee0b90007b1d821fb664ff6784d65f20f9e1203a72b6c68504f2ee5d615461e"} Nov 02 23:15:01.343559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:01.343534 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" event={"ID":"97e91272-bc78-41ad-b139-b28c0f518756","Type":"ContainerStarted","Data":"5d2209e6ab456b8a232a176ee98b845a173629c15bf5badea88db56ee0ece61a"} Nov 02 23:15:02.353330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.353301 5672 generic.go:356] "Generic (PLEG): container finished" podID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerID="9765c38b55ca7bcbbc2a2c0df5c218e6d21487bdfc096ce2046654dcd8b9c1c8" exitCode=60 Nov 02 23:15:02.353727 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.353386 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" event={"ID":"28f50d8f-82bb-4fcc-ac57-6085d5916f5e","Type":"ContainerDied","Data":"9765c38b55ca7bcbbc2a2c0df5c218e6d21487bdfc096ce2046654dcd8b9c1c8"} Nov 02 23:15:02.353727 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.353711 5672 scope.go:122] "RemoveContainer" containerID="9765c38b55ca7bcbbc2a2c0df5c218e6d21487bdfc096ce2046654dcd8b9c1c8" Nov 02 23:15:02.355499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.355471 5672 generic.go:356] "Generic (PLEG): container finished" podID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerID="0c25f15222f14f6e0d27436922e9d17486e8e7da8ef1a2cf1523a5020e390dab" exitCode=60 Nov 02 23:15:02.355894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.355873 5672 scope.go:122] "RemoveContainer" containerID="0c25f15222f14f6e0d27436922e9d17486e8e7da8ef1a2cf1523a5020e390dab" Nov 02 23:15:02.355980 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.355912 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" event={"ID":"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693","Type":"ContainerDied","Data":"0c25f15222f14f6e0d27436922e9d17486e8e7da8ef1a2cf1523a5020e390dab"} Nov 02 23:15:02.369660 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.369631 5672 generic.go:356] "Generic (PLEG): container finished" podID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerID="af5b0efe84fe8cb3d873db3da2d30021bfc879985c5f266c97f5cfe86f70ef29" exitCode=60 Nov 02 23:15:02.369783 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.369742 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" event={"ID":"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575","Type":"ContainerDied","Data":"af5b0efe84fe8cb3d873db3da2d30021bfc879985c5f266c97f5cfe86f70ef29"} Nov 02 23:15:02.370047 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.370016 5672 scope.go:122] "RemoveContainer" containerID="af5b0efe84fe8cb3d873db3da2d30021bfc879985c5f266c97f5cfe86f70ef29" Nov 02 23:15:02.375652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.375626 5672 generic.go:356] "Generic (PLEG): container finished" podID="0632195a-41cb-4195-98a7-feb73e326acf" containerID="122aa1618c5d68efa646a704d69bc1eacfc84c7312fbc49637e28b5da2a8033f" exitCode=60 Nov 02 23:15:02.375739 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.375713 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" event={"ID":"0632195a-41cb-4195-98a7-feb73e326acf","Type":"ContainerDied","Data":"122aa1618c5d68efa646a704d69bc1eacfc84c7312fbc49637e28b5da2a8033f"} Nov 02 23:15:02.375972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.375958 5672 scope.go:122] "RemoveContainer" containerID="122aa1618c5d68efa646a704d69bc1eacfc84c7312fbc49637e28b5da2a8033f" Nov 02 23:15:02.377417 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.377395 5672 generic.go:356] "Generic (PLEG): container finished" podID="97e91272-bc78-41ad-b139-b28c0f518756" containerID="6025bd9ce6320a586761d09df09d8c3f237595a06d5026a08002ce414f6c74e2" exitCode=60 Nov 02 23:15:02.377521 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.377466 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" event={"ID":"97e91272-bc78-41ad-b139-b28c0f518756","Type":"ContainerDied","Data":"6025bd9ce6320a586761d09df09d8c3f237595a06d5026a08002ce414f6c74e2"} Nov 02 23:15:02.377640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:02.377627 5672 scope.go:122] "RemoveContainer" containerID="6025bd9ce6320a586761d09df09d8c3f237595a06d5026a08002ce414f6c74e2" Nov 02 23:15:03.423233 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:03.422163 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" event={"ID":"0632195a-41cb-4195-98a7-feb73e326acf","Type":"ContainerStarted","Data":"b4d4a98458ccc33b2d616892647c6bdd111a8a1255cb27d96681e5a59ce14445"} Nov 02 23:15:03.459992 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:03.459964 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" event={"ID":"97e91272-bc78-41ad-b139-b28c0f518756","Type":"ContainerStarted","Data":"aa1fe87198311c6a63ea29590292fd3c7303ddfc58ace34c37012ce2cd88088d"} Nov 02 23:15:03.469170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:03.468531 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" event={"ID":"28f50d8f-82bb-4fcc-ac57-6085d5916f5e","Type":"ContainerStarted","Data":"0ee6fa039a6da007479f1c8b29c683dd773124167a9d42084f716a81b9a0f64f"} Nov 02 23:15:03.472484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:03.472450 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" event={"ID":"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693","Type":"ContainerStarted","Data":"f1f06aac26bba4ba5764a3ea4be5eecdad0cb9b3db3dc8abd4acaf9cac7a8f9c"} Nov 02 23:15:03.477050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:03.476393 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" event={"ID":"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575","Type":"ContainerStarted","Data":"83e52e88e0b7d92be7bdfce31f1a3a0566bcadaa23bd3cc3d3267419ea201e52"} Nov 02 23:15:03.544994 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:03.544938 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" podStartSLOduration=3.544923264 podStartE2EDuration="3.544923264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:03.540899117 +0000 UTC m=+1296.948210946" watchObservedRunningTime="2026-11-02 23:15:03.544923264 +0000 UTC m=+1296.952235093" Nov 02 23:15:03.650056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:03.649953 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" podStartSLOduration=3.649933485 podStartE2EDuration="3.649933485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:03.646620222 +0000 UTC m=+1297.053932053" watchObservedRunningTime="2026-11-02 23:15:03.649933485 +0000 UTC m=+1297.057245317" Nov 02 23:15:03.754380 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:03.754332 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" podStartSLOduration=3.754318306 podStartE2EDuration="3.754318306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:03.751996822 +0000 UTC m=+1297.159308652" watchObservedRunningTime="2026-11-02 23:15:03.754318306 +0000 UTC m=+1297.161630146" Nov 02 23:15:03.860507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:03.860447 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" podStartSLOduration=3.860428578 podStartE2EDuration="3.860428578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:03.858391282 +0000 UTC m=+1297.265703112" watchObservedRunningTime="2026-11-02 23:15:03.860428578 +0000 UTC m=+1297.267740409" Nov 02 23:15:03.965552 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:03.965495 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" podStartSLOduration=3.965483218 podStartE2EDuration="3.965483218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:03.964133371 +0000 UTC m=+1297.371445187" watchObservedRunningTime="2026-11-02 23:15:03.965483218 +0000 UTC m=+1297.372795048" Nov 02 23:15:04.487128 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.487092 5672 generic.go:356] "Generic (PLEG): container finished" podID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerID="0ee6fa039a6da007479f1c8b29c683dd773124167a9d42084f716a81b9a0f64f" exitCode=60 Nov 02 23:15:04.487592 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.487179 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" event={"ID":"28f50d8f-82bb-4fcc-ac57-6085d5916f5e","Type":"ContainerDied","Data":"0ee6fa039a6da007479f1c8b29c683dd773124167a9d42084f716a81b9a0f64f"} Nov 02 23:15:04.487592 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.487238 5672 scope.go:122] "RemoveContainer" containerID="9765c38b55ca7bcbbc2a2c0df5c218e6d21487bdfc096ce2046654dcd8b9c1c8" Nov 02 23:15:04.487592 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.487579 5672 scope.go:122] "RemoveContainer" containerID="0ee6fa039a6da007479f1c8b29c683dd773124167a9d42084f716a81b9a0f64f" Nov 02 23:15:04.487924 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:04.487899 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894355-27r8g_odh-ai-gateway-infra(28f50d8f-82bb-4fcc-ac57-6085d5916f5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" Nov 02 23:15:04.489469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.489449 5672 generic.go:356] "Generic (PLEG): container finished" podID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerID="f1f06aac26bba4ba5764a3ea4be5eecdad0cb9b3db3dc8abd4acaf9cac7a8f9c" exitCode=60 Nov 02 23:15:04.489581 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.489514 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" event={"ID":"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693","Type":"ContainerDied","Data":"f1f06aac26bba4ba5764a3ea4be5eecdad0cb9b3db3dc8abd4acaf9cac7a8f9c"} Nov 02 23:15:04.489779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.489760 5672 scope.go:122] "RemoveContainer" containerID="f1f06aac26bba4ba5764a3ea4be5eecdad0cb9b3db3dc8abd4acaf9cac7a8f9c" Nov 02 23:15:04.490033 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:04.490010 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q_odh-ai-gateway-infra(502911d5-6f2b-4dd6-8f5b-8fe9c9b50693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" Nov 02 23:15:04.491406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.491386 5672 generic.go:356] "Generic (PLEG): container finished" podID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerID="83e52e88e0b7d92be7bdfce31f1a3a0566bcadaa23bd3cc3d3267419ea201e52" exitCode=60 Nov 02 23:15:04.491514 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.491471 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" event={"ID":"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575","Type":"ContainerDied","Data":"83e52e88e0b7d92be7bdfce31f1a3a0566bcadaa23bd3cc3d3267419ea201e52"} Nov 02 23:15:04.491652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.491638 5672 scope.go:122] "RemoveContainer" containerID="83e52e88e0b7d92be7bdfce31f1a3a0566bcadaa23bd3cc3d3267419ea201e52" Nov 02 23:15:04.491881 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:04.491864 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87_odh-ai-gateway-infra(6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" Nov 02 23:15:04.493577 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.493554 5672 generic.go:356] "Generic (PLEG): container finished" podID="0632195a-41cb-4195-98a7-feb73e326acf" containerID="b4d4a98458ccc33b2d616892647c6bdd111a8a1255cb27d96681e5a59ce14445" exitCode=60 Nov 02 23:15:04.493665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.493639 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" event={"ID":"0632195a-41cb-4195-98a7-feb73e326acf","Type":"ContainerDied","Data":"b4d4a98458ccc33b2d616892647c6bdd111a8a1255cb27d96681e5a59ce14445"} Nov 02 23:15:04.493820 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.493804 5672 scope.go:122] "RemoveContainer" containerID="b4d4a98458ccc33b2d616892647c6bdd111a8a1255cb27d96681e5a59ce14445" Nov 02 23:15:04.494061 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:04.494037 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl_odh-ai-gateway-infra(0632195a-41cb-4195-98a7-feb73e326acf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" podUID="0632195a-41cb-4195-98a7-feb73e326acf" Nov 02 23:15:04.495590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.495569 5672 generic.go:356] "Generic (PLEG): container finished" podID="97e91272-bc78-41ad-b139-b28c0f518756" containerID="aa1fe87198311c6a63ea29590292fd3c7303ddfc58ace34c37012ce2cd88088d" exitCode=60 Nov 02 23:15:04.495671 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.495651 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" event={"ID":"97e91272-bc78-41ad-b139-b28c0f518756","Type":"ContainerDied","Data":"aa1fe87198311c6a63ea29590292fd3c7303ddfc58ace34c37012ce2cd88088d"} Nov 02 23:15:04.495786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.495773 5672 scope.go:122] "RemoveContainer" containerID="aa1fe87198311c6a63ea29590292fd3c7303ddfc58ace34c37012ce2cd88088d" Nov 02 23:15:04.495984 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:04.495968 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz_odh-ai-gateway-infra(97e91272-bc78-41ad-b139-b28c0f518756)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" podUID="97e91272-bc78-41ad-b139-b28c0f518756" Nov 02 23:15:04.504187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.504139 5672 scope.go:122] "RemoveContainer" containerID="0c25f15222f14f6e0d27436922e9d17486e8e7da8ef1a2cf1523a5020e390dab" Nov 02 23:15:04.516605 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.516586 5672 scope.go:122] "RemoveContainer" containerID="af5b0efe84fe8cb3d873db3da2d30021bfc879985c5f266c97f5cfe86f70ef29" Nov 02 23:15:04.527633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.527616 5672 scope.go:122] "RemoveContainer" containerID="122aa1618c5d68efa646a704d69bc1eacfc84c7312fbc49637e28b5da2a8033f" Nov 02 23:15:04.539709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:04.539691 5672 scope.go:122] "RemoveContainer" containerID="6025bd9ce6320a586761d09df09d8c3f237595a06d5026a08002ce414f6c74e2" Nov 02 23:15:05.508012 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:05.507985 5672 scope.go:122] "RemoveContainer" containerID="0ee6fa039a6da007479f1c8b29c683dd773124167a9d42084f716a81b9a0f64f" Nov 02 23:15:05.508507 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:05.508338 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894355-27r8g_odh-ai-gateway-infra(28f50d8f-82bb-4fcc-ac57-6085d5916f5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" Nov 02 23:15:05.510050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:05.510029 5672 scope.go:122] "RemoveContainer" containerID="f1f06aac26bba4ba5764a3ea4be5eecdad0cb9b3db3dc8abd4acaf9cac7a8f9c" Nov 02 23:15:05.510367 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:05.510353 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q_odh-ai-gateway-infra(502911d5-6f2b-4dd6-8f5b-8fe9c9b50693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" Nov 02 23:15:05.511975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:05.511956 5672 scope.go:122] "RemoveContainer" containerID="83e52e88e0b7d92be7bdfce31f1a3a0566bcadaa23bd3cc3d3267419ea201e52" Nov 02 23:15:05.512222 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:05.512198 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87_odh-ai-gateway-infra(6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" Nov 02 23:15:05.513748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:05.513728 5672 scope.go:122] "RemoveContainer" containerID="b4d4a98458ccc33b2d616892647c6bdd111a8a1255cb27d96681e5a59ce14445" Nov 02 23:15:05.513960 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:05.513938 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl_odh-ai-gateway-infra(0632195a-41cb-4195-98a7-feb73e326acf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" podUID="0632195a-41cb-4195-98a7-feb73e326acf" Nov 02 23:15:05.515456 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:05.515437 5672 scope.go:122] "RemoveContainer" containerID="aa1fe87198311c6a63ea29590292fd3c7303ddfc58ace34c37012ce2cd88088d" Nov 02 23:15:05.515664 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:05.515643 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz_odh-ai-gateway-infra(97e91272-bc78-41ad-b139-b28c0f518756)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" podUID="97e91272-bc78-41ad-b139-b28c0f518756" Nov 02 23:15:09.108378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.108343 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc"] Nov 02 23:15:09.112761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.112740 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.208132 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.208102 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e953257b\"/\"kube-root-ca.crt\"" Nov 02 23:15:09.208132 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.208120 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e953257b\"/\"default-dockercfg-mt9rw\"" Nov 02 23:15:09.208752 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.208179 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e953257b\"/\"openshift-service-ca.crt\"" Nov 02 23:15:09.208752 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.208208 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e953257b\"/\"e2e-lifecycle-model-e953257b-kserve-self-signed-certs\"" Nov 02 23:15:09.208752 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.208705 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-dshm\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.208888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.208757 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-tmp-dir\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.208888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.208857 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2404095a-13ed-45fc-8470-79dc7da99cd7-tls-certs\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.209092 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.209067 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-model-cache\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.209260 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.209238 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57g2t\" (UniqueName: \"kubernetes.io/projected/2404095a-13ed-45fc-8470-79dc7da99cd7-kube-api-access-57g2t\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.209373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.209345 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-home\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.213419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.213393 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc"] Nov 02 23:15:09.310488 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.310449 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-model-cache\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.310662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.310571 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57g2t\" (UniqueName: \"kubernetes.io/projected/2404095a-13ed-45fc-8470-79dc7da99cd7-kube-api-access-57g2t\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.310662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.310654 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-home\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.310794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.310699 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-dshm\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.310794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.310742 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-tmp-dir\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.310794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.310779 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2404095a-13ed-45fc-8470-79dc7da99cd7-tls-certs\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.310951 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.310903 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-model-cache\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.311200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.311181 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-home\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.311460 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.311439 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-tmp-dir\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.313774 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.313746 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-dshm\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.313915 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.313895 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2404095a-13ed-45fc-8470-79dc7da99cd7-tls-certs\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.366395 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.366317 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57g2t\" (UniqueName: \"kubernetes.io/projected/2404095a-13ed-45fc-8470-79dc7da99cd7-kube-api-access-57g2t\") pod \"e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.427171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.425128 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:09.811379 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:15:09.811347 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2404095a_13ed_45fc_8470_79dc7da99cd7.slice/crio-76080c20829fcc0829f7689f974c5fb6ff4f11a793d3770e70369a692410cd66 WatchSource:0}: Error finding container 76080c20829fcc0829f7689f974c5fb6ff4f11a793d3770e70369a692410cd66: Status 404 returned error can't find the container with id 76080c20829fcc0829f7689f974c5fb6ff4f11a793d3770e70369a692410cd66 Nov 02 23:15:09.813715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:09.813689 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc"] Nov 02 23:15:10.591072 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:10.591037 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" event={"ID":"2404095a-13ed-45fc-8470-79dc7da99cd7","Type":"ContainerStarted","Data":"5630a83f58a277089e7e0b582f8de788bd251acdb1bc08e2f0cf2a84d8ae9b35"} Nov 02 23:15:10.591479 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:10.591085 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" event={"ID":"2404095a-13ed-45fc-8470-79dc7da99cd7","Type":"ContainerStarted","Data":"76080c20829fcc0829f7689f974c5fb6ff4f11a793d3770e70369a692410cd66"} Nov 02 23:15:10.726483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:10.726429 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" podStartSLOduration=1.726414751 podStartE2EDuration="1.726414751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:10.700397762 +0000 UTC m=+1304.107709593" watchObservedRunningTime="2026-11-02 23:15:10.726414751 +0000 UTC m=+1304.133726605" Nov 02 23:15:11.425765 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:11.425728 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:14.706296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:14.706258 5672 scope.go:122] "RemoveContainer" containerID="0ee6fa039a6da007479f1c8b29c683dd773124167a9d42084f716a81b9a0f64f" Nov 02 23:15:14.706817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:14.706590 5672 scope.go:122] "RemoveContainer" containerID="b4d4a98458ccc33b2d616892647c6bdd111a8a1255cb27d96681e5a59ce14445" Nov 02 23:15:14.707224 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:14.707174 5672 scope.go:122] "RemoveContainer" containerID="f1f06aac26bba4ba5764a3ea4be5eecdad0cb9b3db3dc8abd4acaf9cac7a8f9c" Nov 02 23:15:15.632495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:15.632452 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" event={"ID":"28f50d8f-82bb-4fcc-ac57-6085d5916f5e","Type":"ContainerStarted","Data":"736b38f4bd261b4e1f18506acf1305a5aea69ca7c195f8149b6708cba5cffc27"} Nov 02 23:15:15.634987 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:15.634958 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" event={"ID":"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693","Type":"ContainerStarted","Data":"5c80a818c9df40aafd128a15095b6926362634adf7b4b386de972415ecdd5456"} Nov 02 23:15:15.637206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:15.637176 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" event={"ID":"0632195a-41cb-4195-98a7-feb73e326acf","Type":"ContainerStarted","Data":"6c18ba3adf2723499ddaa4a58a2ea1f75c161bd0678f7d737de4b83628f7219e"} Nov 02 23:15:15.706289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:15.706254 5672 scope.go:122] "RemoveContainer" containerID="aa1fe87198311c6a63ea29590292fd3c7303ddfc58ace34c37012ce2cd88088d" Nov 02 23:15:16.645965 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.645930 5672 generic.go:356] "Generic (PLEG): container finished" podID="0632195a-41cb-4195-98a7-feb73e326acf" containerID="6c18ba3adf2723499ddaa4a58a2ea1f75c161bd0678f7d737de4b83628f7219e" exitCode=60 Nov 02 23:15:16.646489 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.646002 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" event={"ID":"0632195a-41cb-4195-98a7-feb73e326acf","Type":"ContainerDied","Data":"6c18ba3adf2723499ddaa4a58a2ea1f75c161bd0678f7d737de4b83628f7219e"} Nov 02 23:15:16.646489 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.646061 5672 scope.go:122] "RemoveContainer" containerID="b4d4a98458ccc33b2d616892647c6bdd111a8a1255cb27d96681e5a59ce14445" Nov 02 23:15:16.646624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.646528 5672 scope.go:122] "RemoveContainer" containerID="6c18ba3adf2723499ddaa4a58a2ea1f75c161bd0678f7d737de4b83628f7219e" Nov 02 23:15:16.646883 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:16.646856 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl_odh-ai-gateway-infra(0632195a-41cb-4195-98a7-feb73e326acf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" podUID="0632195a-41cb-4195-98a7-feb73e326acf" Nov 02 23:15:16.648905 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.648876 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" event={"ID":"97e91272-bc78-41ad-b139-b28c0f518756","Type":"ContainerStarted","Data":"9db3e9e83f10f44135235bf87da121c2d3c4d8545dddfe5a42516129724a76e9"} Nov 02 23:15:16.650966 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.650948 5672 generic.go:356] "Generic (PLEG): container finished" podID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerID="736b38f4bd261b4e1f18506acf1305a5aea69ca7c195f8149b6708cba5cffc27" exitCode=60 Nov 02 23:15:16.651067 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.651030 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" event={"ID":"28f50d8f-82bb-4fcc-ac57-6085d5916f5e","Type":"ContainerDied","Data":"736b38f4bd261b4e1f18506acf1305a5aea69ca7c195f8149b6708cba5cffc27"} Nov 02 23:15:16.651377 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.651362 5672 scope.go:122] "RemoveContainer" containerID="736b38f4bd261b4e1f18506acf1305a5aea69ca7c195f8149b6708cba5cffc27" Nov 02 23:15:16.651660 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:16.651634 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894355-27r8g_odh-ai-gateway-infra(28f50d8f-82bb-4fcc-ac57-6085d5916f5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" Nov 02 23:15:16.653131 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.653108 5672 generic.go:356] "Generic (PLEG): container finished" podID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerID="5c80a818c9df40aafd128a15095b6926362634adf7b4b386de972415ecdd5456" exitCode=60 Nov 02 23:15:16.653231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.653160 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" event={"ID":"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693","Type":"ContainerDied","Data":"5c80a818c9df40aafd128a15095b6926362634adf7b4b386de972415ecdd5456"} Nov 02 23:15:16.653501 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.653487 5672 scope.go:122] "RemoveContainer" containerID="5c80a818c9df40aafd128a15095b6926362634adf7b4b386de972415ecdd5456" Nov 02 23:15:16.653723 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:16.653707 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q_odh-ai-gateway-infra(502911d5-6f2b-4dd6-8f5b-8fe9c9b50693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" Nov 02 23:15:16.659259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.659242 5672 scope.go:122] "RemoveContainer" containerID="0ee6fa039a6da007479f1c8b29c683dd773124167a9d42084f716a81b9a0f64f" Nov 02 23:15:16.672532 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.672512 5672 scope.go:122] "RemoveContainer" containerID="f1f06aac26bba4ba5764a3ea4be5eecdad0cb9b3db3dc8abd4acaf9cac7a8f9c" Nov 02 23:15:16.755631 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.755601 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g"] Nov 02 23:15:16.863224 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.863192 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl"] Nov 02 23:15:16.976201 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:16.976163 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q"] Nov 02 23:15:17.677285 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:17.677250 5672 generic.go:356] "Generic (PLEG): container finished" podID="97e91272-bc78-41ad-b139-b28c0f518756" containerID="9db3e9e83f10f44135235bf87da121c2d3c4d8545dddfe5a42516129724a76e9" exitCode=60 Nov 02 23:15:17.677285 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:17.677274 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" event={"ID":"97e91272-bc78-41ad-b139-b28c0f518756","Type":"ContainerDied","Data":"9db3e9e83f10f44135235bf87da121c2d3c4d8545dddfe5a42516129724a76e9"} Nov 02 23:15:17.677832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:17.677354 5672 scope.go:122] "RemoveContainer" containerID="aa1fe87198311c6a63ea29590292fd3c7303ddfc58ace34c37012ce2cd88088d" Nov 02 23:15:17.677832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:17.677684 5672 scope.go:122] "RemoveContainer" containerID="9db3e9e83f10f44135235bf87da121c2d3c4d8545dddfe5a42516129724a76e9" Nov 02 23:15:17.678034 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:17.678014 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz_odh-ai-gateway-infra(97e91272-bc78-41ad-b139-b28c0f518756)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" podUID="97e91272-bc78-41ad-b139-b28c0f518756" Nov 02 23:15:17.706886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:17.706850 5672 scope.go:122] "RemoveContainer" containerID="83e52e88e0b7d92be7bdfce31f1a3a0566bcadaa23bd3cc3d3267419ea201e52" Nov 02 23:15:17.881255 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:17.881225 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz"] Nov 02 23:15:17.989975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:17.989947 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" Nov 02 23:15:17.992980 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:17.992953 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" Nov 02 23:15:17.996377 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:17.996355 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" Nov 02 23:15:18.130483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.130444 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0632195a-41cb-4195-98a7-feb73e326acf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0632195a-41cb-4195-98a7-feb73e326acf-maas-api-service-ca\") pod \"0632195a-41cb-4195-98a7-feb73e326acf\" (UID: \"0632195a-41cb-4195-98a7-feb73e326acf\") " Nov 02 23:15:18.130658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.130527 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-kube-api-access-p9mtq\" (UniqueName: \"kubernetes.io/projected/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-kube-api-access-p9mtq\") pod \"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693\" (UID: \"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693\") " Nov 02 23:15:18.130658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.130599 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0632195a-41cb-4195-98a7-feb73e326acf-kube-api-access-hd7lb\" (UniqueName: \"kubernetes.io/projected/0632195a-41cb-4195-98a7-feb73e326acf-kube-api-access-hd7lb\") pod \"0632195a-41cb-4195-98a7-feb73e326acf\" (UID: \"0632195a-41cb-4195-98a7-feb73e326acf\") " Nov 02 23:15:18.130809 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.130669 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-maas-api-service-ca\") pod \"28f50d8f-82bb-4fcc-ac57-6085d5916f5e\" (UID: \"28f50d8f-82bb-4fcc-ac57-6085d5916f5e\") " Nov 02 23:15:18.130862 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.130843 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-maas-api-service-ca\") pod \"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693\" (UID: \"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693\") " Nov 02 23:15:18.130956 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.130931 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-kube-api-access-8np87\" (UniqueName: \"kubernetes.io/projected/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-kube-api-access-8np87\") pod \"28f50d8f-82bb-4fcc-ac57-6085d5916f5e\" (UID: \"28f50d8f-82bb-4fcc-ac57-6085d5916f5e\") " Nov 02 23:15:18.131088 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.131060 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0632195a-41cb-4195-98a7-feb73e326acf-maas-api-service-ca" pod "0632195a-41cb-4195-98a7-feb73e326acf" (UID: "0632195a-41cb-4195-98a7-feb73e326acf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:18.131521 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.131496 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0632195a-41cb-4195-98a7-feb73e326acf-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.131792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.131770 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-maas-api-service-ca" pod "28f50d8f-82bb-4fcc-ac57-6085d5916f5e" (UID: "28f50d8f-82bb-4fcc-ac57-6085d5916f5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:18.132579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.132551 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-maas-api-service-ca" pod "502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" (UID: "502911d5-6f2b-4dd6-8f5b-8fe9c9b50693"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:18.146897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.133519 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-kube-api-access-p9mtq" pod "502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" (UID: "502911d5-6f2b-4dd6-8f5b-8fe9c9b50693"). InnerVolumeSpecName "kube-api-access-p9mtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:18.146897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.134446 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0632195a-41cb-4195-98a7-feb73e326acf-kube-api-access-hd7lb" pod "0632195a-41cb-4195-98a7-feb73e326acf" (UID: "0632195a-41cb-4195-98a7-feb73e326acf"). InnerVolumeSpecName "kube-api-access-hd7lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:18.157194 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.155193 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-kube-api-access-8np87" pod "28f50d8f-82bb-4fcc-ac57-6085d5916f5e" (UID: "28f50d8f-82bb-4fcc-ac57-6085d5916f5e"). InnerVolumeSpecName "kube-api-access-8np87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:18.241594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.239035 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8np87\" (UniqueName: \"kubernetes.io/projected/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-kube-api-access-8np87\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.241594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.239076 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9mtq\" (UniqueName: \"kubernetes.io/projected/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-kube-api-access-p9mtq\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.241594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.239094 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd7lb\" (UniqueName: \"kubernetes.io/projected/0632195a-41cb-4195-98a7-feb73e326acf-kube-api-access-hd7lb\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.241594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.239108 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f50d8f-82bb-4fcc-ac57-6085d5916f5e-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.241594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.239122 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:15:18.695938 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.695899 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" event={"ID":"28f50d8f-82bb-4fcc-ac57-6085d5916f5e","Type":"ContainerDied","Data":"82be05c9d2d392a77a78d605effe936f2c1e22437755bbbc5b1f0b5dab1c4cdd"} Nov 02 23:15:18.696358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.695972 5672 scope.go:122] "RemoveContainer" containerID="736b38f4bd261b4e1f18506acf1305a5aea69ca7c195f8149b6708cba5cffc27" Nov 02 23:15:18.696358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.696062 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g" Nov 02 23:15:18.711395 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.711365 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" Nov 02 23:15:18.721474 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.720122 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q" event={"ID":"502911d5-6f2b-4dd6-8f5b-8fe9c9b50693","Type":"ContainerDied","Data":"0d378ed94a529b75186a5151b8069174c9fd926bafe239d5630ff46c187b90dd"} Nov 02 23:15:18.729222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.722353 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" event={"ID":"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575","Type":"ContainerStarted","Data":"6cc15821de380ce589c5a04f3ba945542762affcceef2b9931ab8bee07bcddd5"} Nov 02 23:15:18.731188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.729547 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" event={"ID":"0632195a-41cb-4195-98a7-feb73e326acf","Type":"ContainerDied","Data":"cee0b90007b1d821fb664ff6784d65f20f9e1203a72b6c68504f2ee5d615461e"} Nov 02 23:15:18.731188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.729648 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl" Nov 02 23:15:18.782821 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.782794 5672 scope.go:122] "RemoveContainer" containerID="5c80a818c9df40aafd128a15095b6926362634adf7b4b386de972415ecdd5456" Nov 02 23:15:18.798348 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.798322 5672 scope.go:122] "RemoveContainer" containerID="6c18ba3adf2723499ddaa4a58a2ea1f75c161bd0678f7d737de4b83628f7219e" Nov 02 23:15:18.909936 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.909907 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q"] Nov 02 23:15:18.914227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.914201 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894355-rqn2q"] Nov 02 23:15:18.924919 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:18.924899 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" Nov 02 23:15:19.014282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.014249 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl"] Nov 02 23:15:19.026011 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.025985 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894355-nx5rl"] Nov 02 23:15:19.057845 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.057816 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e91272-bc78-41ad-b139-b28c0f518756-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e91272-bc78-41ad-b139-b28c0f518756-maas-api-service-ca\") pod \"97e91272-bc78-41ad-b139-b28c0f518756\" (UID: \"97e91272-bc78-41ad-b139-b28c0f518756\") " Nov 02 23:15:19.058001 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.057985 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e91272-bc78-41ad-b139-b28c0f518756-kube-api-access-xbbsq\" (UniqueName: \"kubernetes.io/projected/97e91272-bc78-41ad-b139-b28c0f518756-kube-api-access-xbbsq\") pod \"97e91272-bc78-41ad-b139-b28c0f518756\" (UID: \"97e91272-bc78-41ad-b139-b28c0f518756\") " Nov 02 23:15:19.058291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.058260 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e91272-bc78-41ad-b139-b28c0f518756-maas-api-service-ca" pod "97e91272-bc78-41ad-b139-b28c0f518756" (UID: "97e91272-bc78-41ad-b139-b28c0f518756"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:19.058413 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.058397 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e91272-bc78-41ad-b139-b28c0f518756-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:15:19.060203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.060175 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e91272-bc78-41ad-b139-b28c0f518756-kube-api-access-xbbsq" pod "97e91272-bc78-41ad-b139-b28c0f518756" (UID: "97e91272-bc78-41ad-b139-b28c0f518756"). InnerVolumeSpecName "kube-api-access-xbbsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:19.130001 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.129971 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g"] Nov 02 23:15:19.135744 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.135715 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894355-27r8g"] Nov 02 23:15:19.160114 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.160072 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbbsq\" (UniqueName: \"kubernetes.io/projected/97e91272-bc78-41ad-b139-b28c0f518756-kube-api-access-xbbsq\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:15:19.427434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.425840 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:19.450290 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.450255 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:19.744610 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.744573 5672 generic.go:356] "Generic (PLEG): container finished" podID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerID="6cc15821de380ce589c5a04f3ba945542762affcceef2b9931ab8bee07bcddd5" exitCode=60 Nov 02 23:15:19.745127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.744639 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" event={"ID":"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575","Type":"ContainerDied","Data":"6cc15821de380ce589c5a04f3ba945542762affcceef2b9931ab8bee07bcddd5"} Nov 02 23:15:19.745127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.744692 5672 scope.go:122] "RemoveContainer" containerID="83e52e88e0b7d92be7bdfce31f1a3a0566bcadaa23bd3cc3d3267419ea201e52" Nov 02 23:15:19.745127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.745100 5672 scope.go:122] "RemoveContainer" containerID="6cc15821de380ce589c5a04f3ba945542762affcceef2b9931ab8bee07bcddd5" Nov 02 23:15:19.745863 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:19.745536 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87_odh-ai-gateway-infra(6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" Nov 02 23:15:19.747655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.747636 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" Nov 02 23:15:19.747772 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.747695 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz" event={"ID":"97e91272-bc78-41ad-b139-b28c0f518756","Type":"ContainerDied","Data":"5d2209e6ab456b8a232a176ee98b845a173629c15bf5badea88db56ee0ece61a"} Nov 02 23:15:19.758924 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.758904 5672 scope.go:122] "RemoveContainer" containerID="9db3e9e83f10f44135235bf87da121c2d3c4d8545dddfe5a42516129724a76e9" Nov 02 23:15:19.765589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:19.765567 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:15:20.073051 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:20.072935 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz"] Nov 02 23:15:20.079684 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:20.079649 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894355-9qwjz"] Nov 02 23:15:20.249102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:20.249060 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87"] Nov 02 23:15:20.295996 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:20.295962 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:15:20.713764 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:20.713725 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0632195a-41cb-4195-98a7-feb73e326acf" path="/var/lib/kubelet/pods/0632195a-41cb-4195-98a7-feb73e326acf/volumes" Nov 02 23:15:20.714208 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:20.714189 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" path="/var/lib/kubelet/pods/28f50d8f-82bb-4fcc-ac57-6085d5916f5e/volumes" Nov 02 23:15:20.714640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:20.714618 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" path="/var/lib/kubelet/pods/502911d5-6f2b-4dd6-8f5b-8fe9c9b50693/volumes" Nov 02 23:15:20.715380 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:20.715361 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e91272-bc78-41ad-b139-b28c0f518756" path="/var/lib/kubelet/pods/97e91272-bc78-41ad-b139-b28c0f518756/volumes" Nov 02 23:15:21.071305 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.071266 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" Nov 02 23:15:21.099466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.099424 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-maas-api-service-ca\") pod \"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575\" (UID: \"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575\") " Nov 02 23:15:21.099689 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.099652 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-kube-api-access-s7ccb\" (UniqueName: \"kubernetes.io/projected/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-kube-api-access-s7ccb\") pod \"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575\" (UID: \"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575\") " Nov 02 23:15:21.099831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.099807 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-maas-api-service-ca" pod "6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" (UID: "6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:15:21.100130 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.100109 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:15:21.102084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.102049 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-kube-api-access-s7ccb" pod "6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" (UID: "6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575"). InnerVolumeSpecName "kube-api-access-s7ccb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:15:21.201238 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.201207 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7ccb\" (UniqueName: \"kubernetes.io/projected/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575-kube-api-access-s7ccb\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:15:21.769550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.769523 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" Nov 02 23:15:21.769738 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.769552 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87" event={"ID":"6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575","Type":"ContainerDied","Data":"65086afc20c1e2fa86f80eb0be88e8a155d8dce087ebaa7073ecd6c3d18ef885"} Nov 02 23:15:21.769738 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.769617 5672 scope.go:122] "RemoveContainer" containerID="6cc15821de380ce589c5a04f3ba945542762affcceef2b9931ab8bee07bcddd5" Nov 02 23:15:21.946172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.933857 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87"] Nov 02 23:15:21.946172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:21.941456 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e953257b-29894355-ccp87"] Nov 02 23:15:22.714944 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:22.714912 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" path="/var/lib/kubelet/pods/6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575/volumes" Nov 02 23:15:26.031839 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:26.031791 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:15:26.305298 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305220 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x"] Nov 02 23:15:26.305778 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305764 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0632195a-41cb-4195-98a7-feb73e326acf" containerName="cleanup" Nov 02 23:15:26.305822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305780 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0632195a-41cb-4195-98a7-feb73e326acf" containerName="cleanup" Nov 02 23:15:26.305822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305791 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0632195a-41cb-4195-98a7-feb73e326acf" containerName="cleanup" Nov 02 23:15:26.305822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305796 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0632195a-41cb-4195-98a7-feb73e326acf" containerName="cleanup" Nov 02 23:15:26.305822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305809 5672 cpu_manager.go:414] "containerMap: removing container" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerName="cleanup" Nov 02 23:15:26.305822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305815 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305827 5672 cpu_manager.go:414] "containerMap: removing container" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305833 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305849 5672 cpu_manager.go:414] "containerMap: removing container" podUID="97e91272-bc78-41ad-b139-b28c0f518756" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305854 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e91272-bc78-41ad-b139-b28c0f518756" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305864 5672 cpu_manager.go:414] "containerMap: removing container" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305869 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305878 5672 cpu_manager.go:414] "containerMap: removing container" podUID="97e91272-bc78-41ad-b139-b28c0f518756" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305883 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e91272-bc78-41ad-b139-b28c0f518756" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305891 5672 cpu_manager.go:414] "containerMap: removing container" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305896 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305902 5672 cpu_manager.go:414] "containerMap: removing container" podUID="97e91272-bc78-41ad-b139-b28c0f518756" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305907 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e91272-bc78-41ad-b139-b28c0f518756" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305914 5672 cpu_manager.go:414] "containerMap: removing container" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305918 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305929 5672 cpu_manager.go:414] "containerMap: removing container" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerName="cleanup" Nov 02 23:15:26.305970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.305933 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerName="cleanup" Nov 02 23:15:26.306437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.306007 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerName="cleanup" Nov 02 23:15:26.306437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.306015 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e91272-bc78-41ad-b139-b28c0f518756" containerName="cleanup" Nov 02 23:15:26.306437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.306023 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerName="cleanup" Nov 02 23:15:26.306437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.306030 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e91272-bc78-41ad-b139-b28c0f518756" containerName="cleanup" Nov 02 23:15:26.306437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.306035 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerName="cleanup" Nov 02 23:15:26.306437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.306069 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerName="cleanup" Nov 02 23:15:26.306437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.306074 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerName="cleanup" Nov 02 23:15:26.306437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.306080 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerName="cleanup" Nov 02 23:15:26.306437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.306084 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0632195a-41cb-4195-98a7-feb73e326acf" containerName="cleanup" Nov 02 23:15:26.306437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.306098 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerName="cleanup" Nov 02 23:15:26.306437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.306106 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0632195a-41cb-4195-98a7-feb73e326acf" containerName="cleanup" Nov 02 23:15:26.312865 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.312833 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:15:26.452842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.452791 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f46208-dockercfg-gprbf\"" Nov 02 23:15:26.453258 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.453224 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f46208\"" Nov 02 23:15:26.457119 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.457077 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvc24\" (UniqueName: \"kubernetes.io/projected/2370ad5b-63c5-4296-9f33-ff056ea691ae-kube-api-access-qvc24\") pod \"payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x\" (UID: \"2370ad5b-63c5-4296-9f33-ff056ea691ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:15:26.457322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.457300 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2370ad5b-63c5-4296-9f33-ff056ea691ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x\" (UID: \"2370ad5b-63c5-4296-9f33-ff056ea691ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:15:26.493661 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.493634 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x"] Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.495960 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn"] Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.496940 5672 cpu_manager.go:414] "containerMap: removing container" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerName="cleanup" Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.496961 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerName="cleanup" Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.496978 5672 cpu_manager.go:414] "containerMap: removing container" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerName="cleanup" Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.496986 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerName="cleanup" Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.497041 5672 cpu_manager.go:414] "containerMap: removing container" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerName="cleanup" Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.497049 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f50d8f-82bb-4fcc-ac57-6085d5916f5e" containerName="cleanup" Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.497085 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0632195a-41cb-4195-98a7-feb73e326acf" containerName="cleanup" Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.497093 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0632195a-41cb-4195-98a7-feb73e326acf" containerName="cleanup" Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.497280 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e91272-bc78-41ad-b139-b28c0f518756" containerName="cleanup" Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.497298 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a41fcfc-b6a0-4e8d-b0e2-7e1ba7a5f575" containerName="cleanup" Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.497310 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="502911d5-6f2b-4dd6-8f5b-8fe9c9b50693" containerName="cleanup" Nov 02 23:15:26.498192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.497321 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0632195a-41cb-4195-98a7-feb73e326acf" containerName="cleanup" Nov 02 23:15:26.503134 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.502715 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:15:26.508224 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.508200 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn"] Nov 02 23:15:26.558240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.558134 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvc24\" (UniqueName: \"kubernetes.io/projected/2370ad5b-63c5-4296-9f33-ff056ea691ae-kube-api-access-qvc24\") pod \"payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x\" (UID: \"2370ad5b-63c5-4296-9f33-ff056ea691ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:15:26.558240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.558192 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2370ad5b-63c5-4296-9f33-ff056ea691ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x\" (UID: \"2370ad5b-63c5-4296-9f33-ff056ea691ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:15:26.558481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.558310 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03781ef2-e363-4467-afce-09dc3a4ab0bf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn\" (UID: \"03781ef2-e363-4467-afce-09dc3a4ab0bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:15:26.558481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.558349 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4cs\" (UniqueName: \"kubernetes.io/projected/03781ef2-e363-4467-afce-09dc3a4ab0bf-kube-api-access-9g4cs\") pod \"payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn\" (UID: \"03781ef2-e363-4467-afce-09dc3a4ab0bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:15:26.558906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.558886 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2370ad5b-63c5-4296-9f33-ff056ea691ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x\" (UID: \"2370ad5b-63c5-4296-9f33-ff056ea691ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:15:26.615014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.614982 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvc24\" (UniqueName: \"kubernetes.io/projected/2370ad5b-63c5-4296-9f33-ff056ea691ae-kube-api-access-qvc24\") pod \"payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x\" (UID: \"2370ad5b-63c5-4296-9f33-ff056ea691ae\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:15:26.629969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.629942 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:15:26.659367 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.659335 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03781ef2-e363-4467-afce-09dc3a4ab0bf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn\" (UID: \"03781ef2-e363-4467-afce-09dc3a4ab0bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:15:26.659479 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.659398 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4cs\" (UniqueName: \"kubernetes.io/projected/03781ef2-e363-4467-afce-09dc3a4ab0bf-kube-api-access-9g4cs\") pod \"payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn\" (UID: \"03781ef2-e363-4467-afce-09dc3a4ab0bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:15:26.659999 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.659979 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03781ef2-e363-4467-afce-09dc3a4ab0bf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn\" (UID: \"03781ef2-e363-4467-afce-09dc3a4ab0bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:15:26.713902 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.713867 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4cs\" (UniqueName: \"kubernetes.io/projected/03781ef2-e363-4467-afce-09dc3a4ab0bf-kube-api-access-9g4cs\") pod \"payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn\" (UID: \"03781ef2-e363-4467-afce-09dc3a4ab0bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:15:26.822210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:26.822086 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:15:27.003219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:27.003183 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x"] Nov 02 23:15:27.003517 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:15:27.003479 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2370ad5b_63c5_4296_9f33_ff056ea691ae.slice/crio-8f7d081a03edd2ae0dc5bac3c95790a971244e85e7cbab28f83c89170fc2b399 WatchSource:0}: Error finding container 8f7d081a03edd2ae0dc5bac3c95790a971244e85e7cbab28f83c89170fc2b399: Status 404 returned error can't find the container with id 8f7d081a03edd2ae0dc5bac3c95790a971244e85e7cbab28f83c89170fc2b399 Nov 02 23:15:27.197872 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:27.197844 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn"] Nov 02 23:15:27.197872 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:15:27.197857 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03781ef2_e363_4467_afce_09dc3a4ab0bf.slice/crio-7656bd4f613eb77b3c49ceedae897d80b36cee6138926a323318fb5e0f7a9941 WatchSource:0}: Error finding container 7656bd4f613eb77b3c49ceedae897d80b36cee6138926a323318fb5e0f7a9941: Status 404 returned error can't find the container with id 7656bd4f613eb77b3c49ceedae897d80b36cee6138926a323318fb5e0f7a9941 Nov 02 23:15:27.849393 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:27.849356 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" event={"ID":"03781ef2-e363-4467-afce-09dc3a4ab0bf","Type":"ContainerStarted","Data":"c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75"} Nov 02 23:15:27.849711 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:27.849416 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" event={"ID":"03781ef2-e363-4467-afce-09dc3a4ab0bf","Type":"ContainerStarted","Data":"7656bd4f613eb77b3c49ceedae897d80b36cee6138926a323318fb5e0f7a9941"} Nov 02 23:15:27.849711 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:27.849447 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:15:27.851521 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:27.851493 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" event={"ID":"2370ad5b-63c5-4296-9f33-ff056ea691ae","Type":"ContainerStarted","Data":"deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993"} Nov 02 23:15:27.851655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:27.851532 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" event={"ID":"2370ad5b-63c5-4296-9f33-ff056ea691ae","Type":"ContainerStarted","Data":"8f7d081a03edd2ae0dc5bac3c95790a971244e85e7cbab28f83c89170fc2b399"} Nov 02 23:15:27.851655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:27.851623 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:15:28.003853 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:28.003799 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" podStartSLOduration=2.003787427 podStartE2EDuration="2.003787427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:27.95676979 +0000 UTC m=+1321.364081622" watchObservedRunningTime="2026-11-02 23:15:28.003787427 +0000 UTC m=+1321.411099256" Nov 02 23:15:28.066852 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:28.066794 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" podStartSLOduration=2.066779762 podStartE2EDuration="2.066779762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:28.064877269 +0000 UTC m=+1321.472189098" watchObservedRunningTime="2026-11-02 23:15:28.066779762 +0000 UTC m=+1321.474091591" Nov 02 23:15:33.922667 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:33.922632 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv"] Nov 02 23:15:33.927314 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:33.927276 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.022736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.022701 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-14b321-openshift-default-dockercfg-ssm6b\"" Nov 02 23:15:34.035241 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.028656 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv"] Nov 02 23:15:34.053580 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.053545 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.053734 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.053682 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.053874 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.053845 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fb0d07-d92b-4378-a563-557e881fd75a-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.053929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.053895 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.053990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.053959 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h5lk\" (UniqueName: \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-kube-api-access-9h5lk\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.054045 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.054021 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.054125 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.054107 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fb0d07-d92b-4378-a563-557e881fd75a-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.054193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.054167 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.054374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.054350 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.155676 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.155641 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.155861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.155699 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fb0d07-d92b-4378-a563-557e881fd75a-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.155861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.155735 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.155861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.155812 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.155861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.155839 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.156121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.156080 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.156216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.156191 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.156288 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.156221 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.156288 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.156250 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.156413 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.156328 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fb0d07-d92b-4378-a563-557e881fd75a-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.156413 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.156351 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.156413 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.156366 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h5lk\" (UniqueName: \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-kube-api-access-9h5lk\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.156566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.156525 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fb0d07-d92b-4378-a563-557e881fd75a-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.156794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.156767 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.158665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.158640 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fb0d07-d92b-4378-a563-557e881fd75a-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.158756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.158681 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.209509 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.209445 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.209675 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.209656 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h5lk\" (UniqueName: \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-kube-api-access-9h5lk\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.260063 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.260040 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:34.681904 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.681872 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv"] Nov 02 23:15:34.715297 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:15:34.715268 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fb0d07_d92b_4378_a563_557e881fd75a.slice/crio-4b08c39026c90fc12ae129b4ad7d7933409535039ee7e14845851735016d605a WatchSource:0}: Error finding container 4b08c39026c90fc12ae129b4ad7d7933409535039ee7e14845851735016d605a: Status 404 returned error can't find the container with id 4b08c39026c90fc12ae129b4ad7d7933409535039ee7e14845851735016d605a Nov 02 23:15:34.717930 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.717896 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:15:34.718044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.717977 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:15:34.718044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.718025 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:15:34.907005 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.906960 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" event={"ID":"24fb0d07-d92b-4378-a563-557e881fd75a","Type":"ContainerStarted","Data":"45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9"} Nov 02 23:15:34.907195 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:34.907026 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" event={"ID":"24fb0d07-d92b-4378-a563-557e881fd75a","Type":"ContainerStarted","Data":"4b08c39026c90fc12ae129b4ad7d7933409535039ee7e14845851735016d605a"} Nov 02 23:15:35.022758 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:35.022693 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" podStartSLOduration=2.022678967 podStartE2EDuration="2.022678967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:35.018641263 +0000 UTC m=+1328.425953104" watchObservedRunningTime="2026-11-02 23:15:35.022678967 +0000 UTC m=+1328.429990796" Nov 02 23:15:35.260320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:35.260285 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:35.262179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:35.262113 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 23:15:35.262335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:35.262213 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 23:15:36.260527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:36.260482 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 23:15:36.260970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:36.260538 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 23:15:37.260564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:37.260532 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 23:15:37.260994 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:37.260587 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 23:15:38.260476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:38.260438 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 23:15:38.260664 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:38.260505 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 23:15:38.863978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:38.863950 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:15:38.864173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:38.864012 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:15:39.261114 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:39.261081 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 23:15:39.261619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:39.261143 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 23:15:40.260883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:40.260846 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 23:15:40.261065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:40.260904 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 23:15:41.260445 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:41.260412 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 23:15:41.260938 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:41.260465 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 23:15:42.264906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:42.264875 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:42.265418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:42.265398 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:42.266166 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:15:42.266131 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:15:56.022744 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:15:56.022708 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:16:01.163739 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.163655 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276"] Nov 02 23:16:01.169218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.169192 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:01.221648 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.221259 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276"] Nov 02 23:16:01.262437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.262128 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:16:01.262437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.262288 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:16:01.262437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.262129 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c38777-vd276\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:01.262437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.262358 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:16:01.262765 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.262558 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c38777-vd276\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:01.262765 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.262636 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcfdv\" (UniqueName: \"kubernetes.io/projected/4eb3d538-3424-4b33-98d2-47f9d6903d98-kube-api-access-dcfdv\") pod \"maas-api-revoke-sub-00c38777-vd276\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:01.264143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.264123 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:16:01.363560 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.363522 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c38777-vd276\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:01.363759 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.363593 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcfdv\" (UniqueName: \"kubernetes.io/projected/4eb3d538-3424-4b33-98d2-47f9d6903d98-kube-api-access-dcfdv\") pod \"maas-api-revoke-sub-00c38777-vd276\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:01.363759 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.363630 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c38777-vd276\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:01.364319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.364297 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c38777-vd276\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:01.365819 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.365795 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c38777-vd276\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:01.417692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.417629 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcfdv\" (UniqueName: \"kubernetes.io/projected/4eb3d538-3424-4b33-98d2-47f9d6903d98-kube-api-access-dcfdv\") pod \"maas-api-revoke-sub-00c38777-vd276\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:01.483909 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.483880 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:01.862599 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:16:01.862567 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb3d538_3424_4b33_98d2_47f9d6903d98.slice/crio-78d54544933d10a3568af09f8fd1c293fc2f58a5e33939ca19711a731e0f63e9 WatchSource:0}: Error finding container 78d54544933d10a3568af09f8fd1c293fc2f58a5e33939ca19711a731e0f63e9: Status 404 returned error can't find the container with id 78d54544933d10a3568af09f8fd1c293fc2f58a5e33939ca19711a731e0f63e9 Nov 02 23:16:01.863582 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:01.863550 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276"] Nov 02 23:16:02.177715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:02.177677 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" event={"ID":"4eb3d538-3424-4b33-98d2-47f9d6903d98","Type":"ContainerStarted","Data":"ef2bd1e0d377525373f63140065854516322151b1154a506623d1b19e7b34caa"} Nov 02 23:16:02.178083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:02.177739 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" event={"ID":"4eb3d538-3424-4b33-98d2-47f9d6903d98","Type":"ContainerStarted","Data":"78d54544933d10a3568af09f8fd1c293fc2f58a5e33939ca19711a731e0f63e9"} Nov 02 23:16:02.286416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:02.286363 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" podStartSLOduration=1.286347906 podStartE2EDuration="1.286347906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:02.283642484 +0000 UTC m=+1355.690954326" watchObservedRunningTime="2026-11-02 23:16:02.286347906 +0000 UTC m=+1355.693659735" Nov 02 23:16:03.188017 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:03.187910 5672 generic.go:356] "Generic (PLEG): container finished" podID="4eb3d538-3424-4b33-98d2-47f9d6903d98" containerID="ef2bd1e0d377525373f63140065854516322151b1154a506623d1b19e7b34caa" exitCode=0 Nov 02 23:16:03.188017 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:03.187954 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" event={"ID":"4eb3d538-3424-4b33-98d2-47f9d6903d98","Type":"ContainerDied","Data":"ef2bd1e0d377525373f63140065854516322151b1154a506623d1b19e7b34caa"} Nov 02 23:16:04.045033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.044987 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g"] Nov 02 23:16:04.091276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.091243 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a33334a-f6m6g"] Nov 02 23:16:04.346448 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.346424 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:04.502935 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.502715 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-xks-ca\") pod \"4eb3d538-3424-4b33-98d2-47f9d6903d98\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " Nov 02 23:16:04.504487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.504462 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-openshift-ca\") pod \"4eb3d538-3424-4b33-98d2-47f9d6903d98\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " Nov 02 23:16:04.505287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.505266 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb3d538-3424-4b33-98d2-47f9d6903d98-kube-api-access-dcfdv\" (UniqueName: \"kubernetes.io/projected/4eb3d538-3424-4b33-98d2-47f9d6903d98-kube-api-access-dcfdv\") pod \"4eb3d538-3424-4b33-98d2-47f9d6903d98\" (UID: \"4eb3d538-3424-4b33-98d2-47f9d6903d98\") " Nov 02 23:16:04.505881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.504958 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-openshift-ca" pod "4eb3d538-3424-4b33-98d2-47f9d6903d98" (UID: "4eb3d538-3424-4b33-98d2-47f9d6903d98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:04.511586 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.506639 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-xks-ca" pod "4eb3d538-3424-4b33-98d2-47f9d6903d98" (UID: "4eb3d538-3424-4b33-98d2-47f9d6903d98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:16:04.513197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.513175 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb3d538-3424-4b33-98d2-47f9d6903d98-kube-api-access-dcfdv" pod "4eb3d538-3424-4b33-98d2-47f9d6903d98" (UID: "4eb3d538-3424-4b33-98d2-47f9d6903d98"). InnerVolumeSpecName "kube-api-access-dcfdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:04.612755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.612668 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcfdv\" (UniqueName: \"kubernetes.io/projected/4eb3d538-3424-4b33-98d2-47f9d6903d98-kube-api-access-dcfdv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:16:04.612755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.612708 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:16:04.612755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.612724 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3d538-3424-4b33-98d2-47f9d6903d98-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:16:04.740507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:04.740464 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94e21cd-1b1a-4972-8d47-18fb8e3c0856" path="/var/lib/kubelet/pods/a94e21cd-1b1a-4972-8d47-18fb8e3c0856/volumes" Nov 02 23:16:05.203925 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:05.203894 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" Nov 02 23:16:05.204098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:05.203926 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276" event={"ID":"4eb3d538-3424-4b33-98d2-47f9d6903d98","Type":"ContainerDied","Data":"78d54544933d10a3568af09f8fd1c293fc2f58a5e33939ca19711a731e0f63e9"} Nov 02 23:16:05.204098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:05.203973 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78d54544933d10a3568af09f8fd1c293fc2f58a5e33939ca19711a731e0f63e9" Nov 02 23:16:19.052469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:19.052378 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw"] Nov 02 23:16:19.098876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:19.098845 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b67ff28-nwvtw"] Nov 02 23:16:20.714259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:20.714228 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5363a3ce-2701-480c-a400-637ad1c33e86" path="/var/lib/kubelet/pods/5363a3ce-2701-480c-a400-637ad1c33e86/volumes" Nov 02 23:16:20.971671 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:20.971636 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc"] Nov 02 23:16:20.972563 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:20.972536 5672 cpu_manager.go:414] "containerMap: removing container" podUID="4eb3d538-3424-4b33-98d2-47f9d6903d98" containerName="revoke-keys" Nov 02 23:16:20.972563 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:20.972563 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb3d538-3424-4b33-98d2-47f9d6903d98" containerName="revoke-keys" Nov 02 23:16:20.972831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:20.972722 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb3d538-3424-4b33-98d2-47f9d6903d98" containerName="revoke-keys" Nov 02 23:16:20.976467 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:20.976442 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:16:21.034851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:21.034806 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc"] Nov 02 23:16:21.071429 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:21.071393 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ede388\"" Nov 02 23:16:21.072061 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:21.072036 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ede388-dockercfg-5wzkp\"" Nov 02 23:16:21.093135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:21.093099 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4fd533-9517-408a-b122-1f3287772b30-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc\" (UID: \"0c4fd533-9517-408a-b122-1f3287772b30\") " pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:16:21.093339 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:21.093245 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wmd\" (UniqueName: \"kubernetes.io/projected/0c4fd533-9517-408a-b122-1f3287772b30-kube-api-access-p8wmd\") pod \"payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc\" (UID: \"0c4fd533-9517-408a-b122-1f3287772b30\") " pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:16:21.194300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:21.194261 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4fd533-9517-408a-b122-1f3287772b30-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc\" (UID: \"0c4fd533-9517-408a-b122-1f3287772b30\") " pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:16:21.194515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:21.194468 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wmd\" (UniqueName: \"kubernetes.io/projected/0c4fd533-9517-408a-b122-1f3287772b30-kube-api-access-p8wmd\") pod \"payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc\" (UID: \"0c4fd533-9517-408a-b122-1f3287772b30\") " pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:16:21.195041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:21.195021 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4fd533-9517-408a-b122-1f3287772b30-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc\" (UID: \"0c4fd533-9517-408a-b122-1f3287772b30\") " pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:16:21.248344 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:21.248267 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wmd\" (UniqueName: \"kubernetes.io/projected/0c4fd533-9517-408a-b122-1f3287772b30-kube-api-access-p8wmd\") pod \"payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc\" (UID: \"0c4fd533-9517-408a-b122-1f3287772b30\") " pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:16:21.297936 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:21.297905 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:16:21.672349 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:21.672319 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc"] Nov 02 23:16:21.673756 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:16:21.673720 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4fd533_9517_408a_b122_1f3287772b30.slice/crio-f8697bcfc51df9a60868e18b99c389a6012c02ee3811132d373d37ae6ccc6850 WatchSource:0}: Error finding container f8697bcfc51df9a60868e18b99c389a6012c02ee3811132d373d37ae6ccc6850: Status 404 returned error can't find the container with id f8697bcfc51df9a60868e18b99c389a6012c02ee3811132d373d37ae6ccc6850 Nov 02 23:16:22.349900 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:22.349863 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" event={"ID":"0c4fd533-9517-408a-b122-1f3287772b30","Type":"ContainerStarted","Data":"5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada"} Nov 02 23:16:22.350350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:22.349930 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" event={"ID":"0c4fd533-9517-408a-b122-1f3287772b30","Type":"ContainerStarted","Data":"f8697bcfc51df9a60868e18b99c389a6012c02ee3811132d373d37ae6ccc6850"} Nov 02 23:16:22.350350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:22.349960 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:16:22.456653 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:22.456604 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" podStartSLOduration=2.456591185 podStartE2EDuration="2.456591185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:22.455614813 +0000 UTC m=+1375.862926641" watchObservedRunningTime="2026-11-02 23:16:22.456591185 +0000 UTC m=+1375.863903013" Nov 02 23:16:22.973422 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:22.973381 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:16:26.083635 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:16:26.083434 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:16:31.049287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:31.049253 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw"] Nov 02 23:16:31.055516 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:31.055487 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17ca43d2-v7hzw"] Nov 02 23:16:32.716534 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:32.716494 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365284f7-cfb6-4a3f-a02d-b69db1a44e15" path="/var/lib/kubelet/pods/365284f7-cfb6-4a3f-a02d-b69db1a44e15/volumes" Nov 02 23:16:33.361131 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:33.361096 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:16:35.011891 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:35.011857 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7415-41c312"] Nov 02 23:16:35.017503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:35.017483 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-41c312" Nov 02 23:16:35.065331 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:35.065302 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-41c312"] Nov 02 23:16:35.105996 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:35.105966 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wbwlb\"" Nov 02 23:16:35.147719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:35.147679 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zgvc\" (UniqueName: \"kubernetes.io/projected/19157133-c00f-4b03-bb3a-c050132457f2-kube-api-access-4zgvc\") pod \"test-curl-7415-41c312\" (UID: \"19157133-c00f-4b03-bb3a-c050132457f2\") " pod="opendatahub/test-curl-7415-41c312" Nov 02 23:16:35.249431 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:35.249395 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgvc\" (UniqueName: \"kubernetes.io/projected/19157133-c00f-4b03-bb3a-c050132457f2-kube-api-access-4zgvc\") pod \"test-curl-7415-41c312\" (UID: \"19157133-c00f-4b03-bb3a-c050132457f2\") " pod="opendatahub/test-curl-7415-41c312" Nov 02 23:16:35.303064 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:35.302984 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zgvc\" (UniqueName: \"kubernetes.io/projected/19157133-c00f-4b03-bb3a-c050132457f2-kube-api-access-4zgvc\") pod \"test-curl-7415-41c312\" (UID: \"19157133-c00f-4b03-bb3a-c050132457f2\") " pod="opendatahub/test-curl-7415-41c312" Nov 02 23:16:35.330175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:35.330131 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-41c312" Nov 02 23:16:35.700133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:35.700102 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7415-41c312"] Nov 02 23:16:35.701001 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:16:35.700968 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19157133_c00f_4b03_bb3a_c050132457f2.slice/crio-4a61d97525ec3e0889aaedebabaf9597ea40a4a2fe3032cb8c02f94c0f5ae8a4 WatchSource:0}: Error finding container 4a61d97525ec3e0889aaedebabaf9597ea40a4a2fe3032cb8c02f94c0f5ae8a4: Status 404 returned error can't find the container with id 4a61d97525ec3e0889aaedebabaf9597ea40a4a2fe3032cb8c02f94c0f5ae8a4 Nov 02 23:16:36.504107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:36.504070 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-41c312" event={"ID":"19157133-c00f-4b03-bb3a-c050132457f2","Type":"ContainerStarted","Data":"4a61d97525ec3e0889aaedebabaf9597ea40a4a2fe3032cb8c02f94c0f5ae8a4"} Nov 02 23:16:39.534375 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:39.534337 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7415-41c312" event={"ID":"19157133-c00f-4b03-bb3a-c050132457f2","Type":"ContainerStarted","Data":"5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6"} Nov 02 23:16:39.640711 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:39.640658 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7415-41c312" podStartSLOduration=2.84120864 podStartE2EDuration="5.640644063s" totalImagesPullingTime="2.799435423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:39.638219505 +0000 UTC m=+1393.045531337" watchObservedRunningTime="2026-11-02 23:16:39.640644063 +0000 UTC m=+1393.047955893" Nov 02 23:16:41.074824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:41.074786 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7415-41c312"] Nov 02 23:16:41.079529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:41.079499 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7415-41c312"] Nov 02 23:16:41.551611 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:41.551555 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7415-41c312" podUID="19157133-c00f-4b03-bb3a-c050132457f2" containerName="test-curl-7415-41c312" containerID="cri-o://5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6" gracePeriod=2 Nov 02 23:16:41.599115 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:16:41.599082 5672 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-41c312\" is forbidden: User \"system:node:ip-10-0-137-4.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-4.us-west-2.compute.internal' and this object" podUID="19157133-c00f-4b03-bb3a-c050132457f2" pod="opendatahub/test-curl-7415-41c312" Nov 02 23:16:41.960748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:41.960659 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h"] Nov 02 23:16:41.961603 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:41.961579 5672 cpu_manager.go:414] "containerMap: removing container" podUID="19157133-c00f-4b03-bb3a-c050132457f2" containerName="test-curl-7415-41c312" Nov 02 23:16:41.961746 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:41.961604 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19157133-c00f-4b03-bb3a-c050132457f2" containerName="test-curl-7415-41c312" Nov 02 23:16:41.961746 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:41.961734 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="19157133-c00f-4b03-bb3a-c050132457f2" containerName="test-curl-7415-41c312" Nov 02 23:16:41.966180 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:41.966158 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:42.014001 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:16:42.013952 5672 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-41c312\" is forbidden: User \"system:node:ip-10-0-137-4.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-4.us-west-2.compute.internal' and this object" podUID="19157133-c00f-4b03-bb3a-c050132457f2" pod="opendatahub/test-curl-7415-41c312" Nov 02 23:16:42.016226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.016197 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h"] Nov 02 23:16:42.054425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.054377 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-xct7h\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:42.054631 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.054474 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-xct7h\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:42.054720 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.054673 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khp45\" (UniqueName: \"kubernetes.io/projected/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-kube-api-access-khp45\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-xct7h\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:42.057686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.057609 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:16:42.057686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.057614 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:16:42.058041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.058007 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:16:42.058305 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.058287 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:16:42.155501 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.155466 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khp45\" (UniqueName: \"kubernetes.io/projected/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-kube-api-access-khp45\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-xct7h\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:42.155962 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.155587 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-xct7h\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:42.155962 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.155685 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-xct7h\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:42.156383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.156359 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-xct7h\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:42.158207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.158181 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-xct7h\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:42.208702 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.208676 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khp45\" (UniqueName: \"kubernetes.io/projected/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-kube-api-access-khp45\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-xct7h\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:42.281494 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.281464 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:42.657177 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:16:42.657113 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d8153e_748b_4cbc_bbbd_61d4f0e08c41.slice/crio-3250f50f202c124cf56e758f3390dd29f9f9a5a1ee38c44f95fc81a03f2c944a WatchSource:0}: Error finding container 3250f50f202c124cf56e758f3390dd29f9f9a5a1ee38c44f95fc81a03f2c944a: Status 404 returned error can't find the container with id 3250f50f202c124cf56e758f3390dd29f9f9a5a1ee38c44f95fc81a03f2c944a Nov 02 23:16:42.658812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:42.658782 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h"] Nov 02 23:16:43.569920 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:43.569889 5672 generic.go:356] "Generic (PLEG): container finished" podID="50d8153e-748b-4cbc-bbbd-61d4f0e08c41" containerID="218049555efe0523d72fdf3be6fc7cb658601f5ada3a3d21111cb651ce8c172b" exitCode=0 Nov 02 23:16:43.570334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:43.569971 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" event={"ID":"50d8153e-748b-4cbc-bbbd-61d4f0e08c41","Type":"ContainerDied","Data":"218049555efe0523d72fdf3be6fc7cb658601f5ada3a3d21111cb651ce8c172b"} Nov 02 23:16:43.570334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:43.570035 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" event={"ID":"50d8153e-748b-4cbc-bbbd-61d4f0e08c41","Type":"ContainerStarted","Data":"3250f50f202c124cf56e758f3390dd29f9f9a5a1ee38c44f95fc81a03f2c944a"} Nov 02 23:16:43.802939 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:43.802911 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-41c312" Nov 02 23:16:43.880722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:43.880690 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19157133-c00f-4b03-bb3a-c050132457f2-kube-api-access-4zgvc\" (UniqueName: \"kubernetes.io/projected/19157133-c00f-4b03-bb3a-c050132457f2-kube-api-access-4zgvc\") pod \"19157133-c00f-4b03-bb3a-c050132457f2\" (UID: \"19157133-c00f-4b03-bb3a-c050132457f2\") " Nov 02 23:16:43.882611 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:43.882585 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19157133-c00f-4b03-bb3a-c050132457f2-kube-api-access-4zgvc" pod "19157133-c00f-4b03-bb3a-c050132457f2" (UID: "19157133-c00f-4b03-bb3a-c050132457f2"). InnerVolumeSpecName "kube-api-access-4zgvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:43.982624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:43.982590 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zgvc\" (UniqueName: \"kubernetes.io/projected/19157133-c00f-4b03-bb3a-c050132457f2-kube-api-access-4zgvc\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:16:44.579022 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.578985 5672 generic.go:356] "Generic (PLEG): container finished" podID="19157133-c00f-4b03-bb3a-c050132457f2" containerID="5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6" exitCode=137 Nov 02 23:16:44.579601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.579044 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7415-41c312" Nov 02 23:16:44.579601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.579084 5672 scope.go:122] "RemoveContainer" containerID="5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6" Nov 02 23:16:44.595312 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.595293 5672 scope.go:122] "RemoveContainer" containerID="5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6" Nov 02 23:16:44.595547 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:16:44.595524 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6\": container with ID starting with 5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6 not found: ID does not exist" containerID="5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6" Nov 02 23:16:44.595634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.595555 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6"} err="failed to get container status \"5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6\": rpc error: code = NotFound desc = could not find container \"5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6\": container with ID starting with 5476b9155d532bb19b2f748b4de677f141212644528499f77686c92fa068fdb6 not found: ID does not exist" Nov 02 23:16:44.636081 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:16:44.636045 5672 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7415-41c312\" is forbidden: User \"system:node:ip-10-0-137-4.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-4.us-west-2.compute.internal' and this object" podUID="19157133-c00f-4b03-bb3a-c050132457f2" pod="opendatahub/test-curl-7415-41c312" Nov 02 23:16:44.716756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.716729 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19157133-c00f-4b03-bb3a-c050132457f2" path="/var/lib/kubelet/pods/19157133-c00f-4b03-bb3a-c050132457f2/volumes" Nov 02 23:16:44.740567 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.740537 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:44.792075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.792044 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-openshift-ca\") pod \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " Nov 02 23:16:44.792251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.792237 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-xks-ca\") pod \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " Nov 02 23:16:44.792357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.792344 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-kube-api-access-khp45\" (UniqueName: \"kubernetes.io/projected/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-kube-api-access-khp45\") pod \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\" (UID: \"50d8153e-748b-4cbc-bbbd-61d4f0e08c41\") " Nov 02 23:16:44.792425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.792398 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-openshift-ca" pod "50d8153e-748b-4cbc-bbbd-61d4f0e08c41" (UID: "50d8153e-748b-4cbc-bbbd-61d4f0e08c41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:44.792652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.792635 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:16:44.794374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.794351 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-xks-ca" pod "50d8153e-748b-4cbc-bbbd-61d4f0e08c41" (UID: "50d8153e-748b-4cbc-bbbd-61d4f0e08c41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:16:44.794476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.794429 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-kube-api-access-khp45" pod "50d8153e-748b-4cbc-bbbd-61d4f0e08c41" (UID: "50d8153e-748b-4cbc-bbbd-61d4f0e08c41"). InnerVolumeSpecName "kube-api-access-khp45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:44.894320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.894231 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khp45\" (UniqueName: \"kubernetes.io/projected/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-kube-api-access-khp45\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:16:44.894320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:44.894260 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d8153e-748b-4cbc-bbbd-61d4f0e08c41-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:16:45.588528 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:45.588499 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" Nov 02 23:16:45.588881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:45.588494 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h" event={"ID":"50d8153e-748b-4cbc-bbbd-61d4f0e08c41","Type":"ContainerDied","Data":"3250f50f202c124cf56e758f3390dd29f9f9a5a1ee38c44f95fc81a03f2c944a"} Nov 02 23:16:45.588881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:45.588624 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3250f50f202c124cf56e758f3390dd29f9f9a5a1ee38c44f95fc81a03f2c944a" Nov 02 23:16:47.045503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:47.045467 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2"] Nov 02 23:16:47.092178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:47.092133 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6ff5280-gjmk2"] Nov 02 23:16:48.721113 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:48.721068 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f0f9b5-393e-49c5-a002-e7f6117cdcf5" path="/var/lib/kubelet/pods/c4f0f9b5-393e-49c5-a002-e7f6117cdcf5/volumes" Nov 02 23:16:50.170689 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.170651 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g"] Nov 02 23:16:50.171503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.171473 5672 cpu_manager.go:414] "containerMap: removing container" podUID="50d8153e-748b-4cbc-bbbd-61d4f0e08c41" containerName="revoke-keys" Nov 02 23:16:50.171503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.171495 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d8153e-748b-4cbc-bbbd-61d4f0e08c41" containerName="revoke-keys" Nov 02 23:16:50.171695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.171620 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d8153e-748b-4cbc-bbbd-61d4f0e08c41" containerName="revoke-keys" Nov 02 23:16:50.175252 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.175231 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:50.226949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.226921 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g"] Nov 02 23:16:50.264744 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.264710 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sxld\" (UniqueName: \"kubernetes.io/projected/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-kube-api-access-6sxld\") pod \"maas-api-revoke-sub-c74b7ec4-gpt6g\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:50.264951 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.264923 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c74b7ec4-gpt6g\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:50.265036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.265019 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c74b7ec4-gpt6g\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:50.268705 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.268674 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:16:50.268705 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.268710 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:16:50.268863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.268696 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:16:50.270996 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.270982 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:16:50.366681 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.366647 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c74b7ec4-gpt6g\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:50.366864 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.366708 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c74b7ec4-gpt6g\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:50.366864 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.366843 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sxld\" (UniqueName: \"kubernetes.io/projected/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-kube-api-access-6sxld\") pod \"maas-api-revoke-sub-c74b7ec4-gpt6g\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:50.367432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.367400 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c74b7ec4-gpt6g\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:50.369182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.369144 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c74b7ec4-gpt6g\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:50.420869 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.420788 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sxld\" (UniqueName: \"kubernetes.io/projected/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-kube-api-access-6sxld\") pod \"maas-api-revoke-sub-c74b7ec4-gpt6g\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:50.489351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.489320 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:50.867225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:50.867190 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g"] Nov 02 23:16:50.868139 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:16:50.868098 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20bcb9e_48bf_4d27_b9a8_5dde3921d1fd.slice/crio-1ab656d2618ffbdc54ee3ec1ea62d421e9eb5c14a3a822770b703ede332bf8aa WatchSource:0}: Error finding container 1ab656d2618ffbdc54ee3ec1ea62d421e9eb5c14a3a822770b703ede332bf8aa: Status 404 returned error can't find the container with id 1ab656d2618ffbdc54ee3ec1ea62d421e9eb5c14a3a822770b703ede332bf8aa Nov 02 23:16:51.637851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:51.637813 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" event={"ID":"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd","Type":"ContainerStarted","Data":"024163a7b5327400c7e3963d8b9811a5f54334a811871d4af668bdd4e42184c0"} Nov 02 23:16:51.638311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:51.637872 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" event={"ID":"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd","Type":"ContainerStarted","Data":"1ab656d2618ffbdc54ee3ec1ea62d421e9eb5c14a3a822770b703ede332bf8aa"} Nov 02 23:16:51.746535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:51.746485 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" podStartSLOduration=1.746470668 podStartE2EDuration="1.746470668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:51.743120832 +0000 UTC m=+1405.150432662" watchObservedRunningTime="2026-11-02 23:16:51.746470668 +0000 UTC m=+1405.153782523" Nov 02 23:16:52.648181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:52.648125 5672 generic.go:356] "Generic (PLEG): container finished" podID="b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd" containerID="024163a7b5327400c7e3963d8b9811a5f54334a811871d4af668bdd4e42184c0" exitCode=0 Nov 02 23:16:52.648689 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:52.648209 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" event={"ID":"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd","Type":"ContainerDied","Data":"024163a7b5327400c7e3963d8b9811a5f54334a811871d4af668bdd4e42184c0"} Nov 02 23:16:53.810834 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:53.810805 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:53.922851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:53.922772 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-openshift-ca\") pod \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " Nov 02 23:16:53.922851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:53.922819 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-xks-ca\") pod \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " Nov 02 23:16:53.923095 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:53.923080 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-kube-api-access-6sxld\" (UniqueName: \"kubernetes.io/projected/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-kube-api-access-6sxld\") pod \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\" (UID: \"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd\") " Nov 02 23:16:53.923198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:53.923179 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-openshift-ca" pod "b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd" (UID: "b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:16:53.923692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:53.923665 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:16:53.925143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:53.925110 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-xks-ca" pod "b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd" (UID: "b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:16:53.929640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:53.929607 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-kube-api-access-6sxld" pod "b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd" (UID: "b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd"). InnerVolumeSpecName "kube-api-access-6sxld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:16:54.025543 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:54.025506 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sxld\" (UniqueName: \"kubernetes.io/projected/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-kube-api-access-6sxld\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:16:54.025543 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:54.025539 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:16:54.665094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:54.665059 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" Nov 02 23:16:54.665295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:54.665097 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g" event={"ID":"b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd","Type":"ContainerDied","Data":"1ab656d2618ffbdc54ee3ec1ea62d421e9eb5c14a3a822770b703ede332bf8aa"} Nov 02 23:16:54.665295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:54.665162 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ab656d2618ffbdc54ee3ec1ea62d421e9eb5c14a3a822770b703ede332bf8aa" Nov 02 23:16:56.032264 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:16:56.032219 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:16:57.057122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:57.057082 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr"] Nov 02 23:16:57.098255 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:57.098220 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a42de9-5tqcr"] Nov 02 23:16:58.713214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:16:58.713181 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="404d1f5c-dc02-480f-ae33-7c8603778e33" path="/var/lib/kubelet/pods/404d1f5c-dc02-480f-ae33-7c8603778e33/volumes" Nov 02 23:17:00.826727 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:00.826692 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl"] Nov 02 23:17:00.927123 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:00.927088 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl"] Nov 02 23:17:03.044629 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:03.044597 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd"] Nov 02 23:17:03.090844 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:03.090810 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c47af0-tlwbd"] Nov 02 23:17:04.715019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:04.714979 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd1ad31-73cb-4aeb-908e-a0ea39a628b5" path="/var/lib/kubelet/pods/dbd1ad31-73cb-4aeb-908e-a0ea39a628b5/volumes" Nov 02 23:17:05.827479 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:05.827408 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" podUID="ed428b35-b9e9-4012-ac33-b9dd5b702d6f" containerName="payload-processing" containerID="cri-o://34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786" gracePeriod=25 Nov 02 23:17:05.927565 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:05.927519 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" podUID="275c4d8b-85fb-48fa-b065-22f798a57ebb" containerName="payload-pre-processing" containerID="cri-o://d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee" gracePeriod=25 Nov 02 23:17:06.114952 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.114918 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:17:06.215432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.215395 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-kube-api-access-ztbcb\" (UniqueName: \"kubernetes.io/projected/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-kube-api-access-ztbcb\") pod \"ed428b35-b9e9-4012-ac33-b9dd5b702d6f\" (UID: \"ed428b35-b9e9-4012-ac33-b9dd5b702d6f\") " Nov 02 23:17:06.215576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.215564 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-plugins-config-volume\") pod \"ed428b35-b9e9-4012-ac33-b9dd5b702d6f\" (UID: \"ed428b35-b9e9-4012-ac33-b9dd5b702d6f\") " Nov 02 23:17:06.216135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.216100 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-plugins-config-volume" pod "ed428b35-b9e9-4012-ac33-b9dd5b702d6f" (UID: "ed428b35-b9e9-4012-ac33-b9dd5b702d6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:06.218198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.218135 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-kube-api-access-ztbcb" pod "ed428b35-b9e9-4012-ac33-b9dd5b702d6f" (UID: "ed428b35-b9e9-4012-ac33-b9dd5b702d6f"). InnerVolumeSpecName "kube-api-access-ztbcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:06.225227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.225205 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:17:06.317182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.317111 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275c4d8b-85fb-48fa-b065-22f798a57ebb-kube-api-access-f58jg\" (UniqueName: \"kubernetes.io/projected/275c4d8b-85fb-48fa-b065-22f798a57ebb-kube-api-access-f58jg\") pod \"275c4d8b-85fb-48fa-b065-22f798a57ebb\" (UID: \"275c4d8b-85fb-48fa-b065-22f798a57ebb\") " Nov 02 23:17:06.317450 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.317429 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275c4d8b-85fb-48fa-b065-22f798a57ebb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275c4d8b-85fb-48fa-b065-22f798a57ebb-plugins-config-volume\") pod \"275c4d8b-85fb-48fa-b065-22f798a57ebb\" (UID: \"275c4d8b-85fb-48fa-b065-22f798a57ebb\") " Nov 02 23:17:06.317787 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.317763 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275c4d8b-85fb-48fa-b065-22f798a57ebb-plugins-config-volume" pod "275c4d8b-85fb-48fa-b065-22f798a57ebb" (UID: "275c4d8b-85fb-48fa-b065-22f798a57ebb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:06.317883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.317867 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztbcb\" (UniqueName: \"kubernetes.io/projected/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-kube-api-access-ztbcb\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:06.317945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.317889 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed428b35-b9e9-4012-ac33-b9dd5b702d6f-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:06.319241 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.319217 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275c4d8b-85fb-48fa-b065-22f798a57ebb-kube-api-access-f58jg" pod "275c4d8b-85fb-48fa-b065-22f798a57ebb" (UID: "275c4d8b-85fb-48fa-b065-22f798a57ebb"). InnerVolumeSpecName "kube-api-access-f58jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:06.418961 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.418872 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275c4d8b-85fb-48fa-b065-22f798a57ebb-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:06.418961 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.418905 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f58jg\" (UniqueName: \"kubernetes.io/projected/275c4d8b-85fb-48fa-b065-22f798a57ebb-kube-api-access-f58jg\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:06.766089 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.766047 5672 generic.go:356] "Generic (PLEG): container finished" podID="ed428b35-b9e9-4012-ac33-b9dd5b702d6f" containerID="34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786" exitCode=0 Nov 02 23:17:06.766268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.766095 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" Nov 02 23:17:06.766268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.766131 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" event={"ID":"ed428b35-b9e9-4012-ac33-b9dd5b702d6f","Type":"ContainerDied","Data":"34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786"} Nov 02 23:17:06.766268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.766203 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl" event={"ID":"ed428b35-b9e9-4012-ac33-b9dd5b702d6f","Type":"ContainerDied","Data":"69a05faf01d9a033e25f4df7c4f673161002f629f5c44ff9c563ad87278b3db2"} Nov 02 23:17:06.766268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.766228 5672 scope.go:122] "RemoveContainer" containerID="34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786" Nov 02 23:17:06.768016 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.767991 5672 generic.go:356] "Generic (PLEG): container finished" podID="275c4d8b-85fb-48fa-b065-22f798a57ebb" containerID="d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee" exitCode=0 Nov 02 23:17:06.768133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.768035 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" event={"ID":"275c4d8b-85fb-48fa-b065-22f798a57ebb","Type":"ContainerDied","Data":"d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee"} Nov 02 23:17:06.768133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.768077 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" event={"ID":"275c4d8b-85fb-48fa-b065-22f798a57ebb","Type":"ContainerDied","Data":"af48c9cd7af00422c9f9dd7eba4fdfc50c748661c5fe5abf4a66ecb4d03b23de"} Nov 02 23:17:06.768133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.768109 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl" Nov 02 23:17:06.777779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.777755 5672 scope.go:122] "RemoveContainer" containerID="34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786" Nov 02 23:17:06.778082 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:17:06.778056 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786\": container with ID starting with 34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786 not found: ID does not exist" containerID="34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786" Nov 02 23:17:06.778165 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.778089 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786"} err="failed to get container status \"34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786\": rpc error: code = NotFound desc = could not find container \"34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786\": container with ID starting with 34b2c515c255dab77839e1e4d5f59828b295879ce4d7b444ff1080402a7b5786 not found: ID does not exist" Nov 02 23:17:06.778165 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.778106 5672 scope.go:122] "RemoveContainer" containerID="d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee" Nov 02 23:17:06.787459 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.787434 5672 scope.go:122] "RemoveContainer" containerID="d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee" Nov 02 23:17:06.787738 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:17:06.787711 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee\": container with ID starting with d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee not found: ID does not exist" containerID="d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee" Nov 02 23:17:06.787809 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.787739 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee"} err="failed to get container status \"d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee\": rpc error: code = NotFound desc = could not find container \"d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee\": container with ID starting with d0ea65a4b6c16ad3c2cb1c0c28134e2c82cbc67411d285c5644a2e9974f588ee not found: ID does not exist" Nov 02 23:17:06.921482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.921451 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl"] Nov 02 23:17:06.925280 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:06.925252 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2a689-6b85648bff-dmctl"] Nov 02 23:17:07.075949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:07.075856 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl"] Nov 02 23:17:07.079021 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:07.078981 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b2a689-69d57cb9bc-j8cgl"] Nov 02 23:17:08.713556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:08.713521 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275c4d8b-85fb-48fa-b065-22f798a57ebb" path="/var/lib/kubelet/pods/275c4d8b-85fb-48fa-b065-22f798a57ebb/volumes" Nov 02 23:17:08.713969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:08.713955 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed428b35-b9e9-4012-ac33-b9dd5b702d6f" path="/var/lib/kubelet/pods/ed428b35-b9e9-4012-ac33-b9dd5b702d6f/volumes" Nov 02 23:17:12.050482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:12.050447 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r"] Nov 02 23:17:12.097210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:12.097173 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e88ea5-x259r"] Nov 02 23:17:12.715191 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:12.715136 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b989047c-5268-4996-888d-aac949c546ec" path="/var/lib/kubelet/pods/b989047c-5268-4996-888d-aac949c546ec/volumes" Nov 02 23:17:13.043518 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:13.043481 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p"] Nov 02 23:17:13.089777 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:13.089743 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f0022f-kkw5p"] Nov 02 23:17:14.713378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:14.713346 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f20596-326f-4689-89cb-5587dfadf663" path="/var/lib/kubelet/pods/b9f20596-326f-4689-89cb-5587dfadf663/volumes" Nov 02 23:17:18.010896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.010855 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq"] Nov 02 23:17:18.011434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.011419 5672 cpu_manager.go:414] "containerMap: removing container" podUID="275c4d8b-85fb-48fa-b065-22f798a57ebb" containerName="payload-pre-processing" Nov 02 23:17:18.011478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.011436 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275c4d8b-85fb-48fa-b065-22f798a57ebb" containerName="payload-pre-processing" Nov 02 23:17:18.011478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.011465 5672 cpu_manager.go:414] "containerMap: removing container" podUID="b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd" containerName="revoke-keys" Nov 02 23:17:18.011478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.011470 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd" containerName="revoke-keys" Nov 02 23:17:18.011576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.011486 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ed428b35-b9e9-4012-ac33-b9dd5b702d6f" containerName="payload-processing" Nov 02 23:17:18.011576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.011491 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed428b35-b9e9-4012-ac33-b9dd5b702d6f" containerName="payload-processing" Nov 02 23:17:18.011576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.011568 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed428b35-b9e9-4012-ac33-b9dd5b702d6f" containerName="payload-processing" Nov 02 23:17:18.011665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.011580 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="275c4d8b-85fb-48fa-b065-22f798a57ebb" containerName="payload-pre-processing" Nov 02 23:17:18.011665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.011591 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd" containerName="revoke-keys" Nov 02 23:17:18.015433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.015412 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:18.065224 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.065192 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:17:18.065515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.065496 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:17:18.111444 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.111408 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:17:18.111699 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.111518 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:17:18.117019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.116986 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq"] Nov 02 23:17:18.192745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.192703 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dbdeeaf-gwkwq\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:18.192921 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.192819 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dbdeeaf-gwkwq\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:18.192971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.192929 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v529\" (UniqueName: \"kubernetes.io/projected/c08ed4a3-d627-4111-9a3e-372d01899e99-kube-api-access-6v529\") pod \"maas-api-revoke-sub-3dbdeeaf-gwkwq\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:18.294784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.294699 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dbdeeaf-gwkwq\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:18.294953 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.294823 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v529\" (UniqueName: \"kubernetes.io/projected/c08ed4a3-d627-4111-9a3e-372d01899e99-kube-api-access-6v529\") pod \"maas-api-revoke-sub-3dbdeeaf-gwkwq\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:18.294953 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.294935 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dbdeeaf-gwkwq\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:18.295423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.295398 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dbdeeaf-gwkwq\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:18.297275 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.297256 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dbdeeaf-gwkwq\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:18.348299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.348268 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v529\" (UniqueName: \"kubernetes.io/projected/c08ed4a3-d627-4111-9a3e-372d01899e99-kube-api-access-6v529\") pod \"maas-api-revoke-sub-3dbdeeaf-gwkwq\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:18.628924 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:18.628847 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:19.005285 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.005254 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq"] Nov 02 23:17:19.005793 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:17:19.005762 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08ed4a3_d627_4111_9a3e_372d01899e99.slice/crio-9df0c95c888968dea7448bd90fa0e3a818b54e4d8ba837783092634c4151316a WatchSource:0}: Error finding container 9df0c95c888968dea7448bd90fa0e3a818b54e4d8ba837783092634c4151316a: Status 404 returned error can't find the container with id 9df0c95c888968dea7448bd90fa0e3a818b54e4d8ba837783092634c4151316a Nov 02 23:17:19.748548 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.748508 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws"] Nov 02 23:17:19.752994 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.752971 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:19.848856 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.848822 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws"] Nov 02 23:17:19.876423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.876386 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" event={"ID":"c08ed4a3-d627-4111-9a3e-372d01899e99","Type":"ContainerStarted","Data":"f5297a64f93827d18646f02cb51bacf7016b651477e559cdb7c69b52eea56bb0"} Nov 02 23:17:19.876672 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.876646 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" event={"ID":"c08ed4a3-d627-4111-9a3e-372d01899e99","Type":"ContainerStarted","Data":"9df0c95c888968dea7448bd90fa0e3a818b54e4d8ba837783092634c4151316a"} Nov 02 23:17:19.919833 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.919797 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:19.919991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.919867 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:19.920036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.919987 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:19.920087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.920038 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:19.920250 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.920232 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgqhp\" (UniqueName: \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-kube-api-access-cgqhp\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:19.920314 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.920294 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:19.920376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.920322 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:19.920416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.920390 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:19.920474 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.920461 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8581a625-fd4b-47d0-9ac1-08f312407c1b-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:19.986327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:19.986269 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" podStartSLOduration=2.9862550690000003 podStartE2EDuration="2.986255069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:19.983724735 +0000 UTC m=+1433.391036566" watchObservedRunningTime="2026-11-02 23:17:19.986255069 +0000 UTC m=+1433.393566896" Nov 02 23:17:20.021441 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.021359 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.021441 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.021396 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.021441 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.021439 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.021728 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.021502 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8581a625-fd4b-47d0-9ac1-08f312407c1b-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.021728 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.021529 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.021728 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.021576 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.021728 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.021651 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.021728 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.021715 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.021979 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.021810 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.021979 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.021858 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.021979 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.021959 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgqhp\" (UniqueName: \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-kube-api-access-cgqhp\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.022139 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.022068 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.022306 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.022269 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8581a625-fd4b-47d0-9ac1-08f312407c1b-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.022306 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.022289 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.023845 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.023825 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.024029 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.023991 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.075222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.075189 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.075352 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.075308 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgqhp\" (UniqueName: \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-kube-api-access-cgqhp\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.368030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.367947 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:20.834967 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:17:20.834929 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8581a625_fd4b_47d0_9ac1_08f312407c1b.slice/crio-2f2569980136b35907ebdd599dbc70d136cd95b54a2bde3545f20a787c659ff1 WatchSource:0}: Error finding container 2f2569980136b35907ebdd599dbc70d136cd95b54a2bde3545f20a787c659ff1: Status 404 returned error can't find the container with id 2f2569980136b35907ebdd599dbc70d136cd95b54a2bde3545f20a787c659ff1 Nov 02 23:17:20.835394 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.835065 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws"] Nov 02 23:17:20.838448 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.838415 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:20.838537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.838481 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:20.838537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.838511 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:20.885330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.885296 5672 generic.go:356] "Generic (PLEG): container finished" podID="c08ed4a3-d627-4111-9a3e-372d01899e99" containerID="f5297a64f93827d18646f02cb51bacf7016b651477e559cdb7c69b52eea56bb0" exitCode=0 Nov 02 23:17:20.885480 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.885370 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" event={"ID":"c08ed4a3-d627-4111-9a3e-372d01899e99","Type":"ContainerDied","Data":"f5297a64f93827d18646f02cb51bacf7016b651477e559cdb7c69b52eea56bb0"} Nov 02 23:17:20.887442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:20.887418 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" event={"ID":"8581a625-fd4b-47d0-9ac1-08f312407c1b","Type":"ContainerStarted","Data":"2f2569980136b35907ebdd599dbc70d136cd95b54a2bde3545f20a787c659ff1"} Nov 02 23:17:21.896194 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:21.896123 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" event={"ID":"8581a625-fd4b-47d0-9ac1-08f312407c1b","Type":"ContainerStarted","Data":"29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a"} Nov 02 23:17:22.066223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.065519 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" podStartSLOduration=3.06549812 podStartE2EDuration="3.06549812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:22.061985972 +0000 UTC m=+1435.469297803" watchObservedRunningTime="2026-11-02 23:17:22.06549812 +0000 UTC m=+1435.472809974" Nov 02 23:17:22.109990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.109960 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:22.264587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.264551 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-openshift-ca\") pod \"c08ed4a3-d627-4111-9a3e-372d01899e99\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " Nov 02 23:17:22.264867 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.264851 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-xks-ca\") pod \"c08ed4a3-d627-4111-9a3e-372d01899e99\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " Nov 02 23:17:22.265135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.265121 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08ed4a3-d627-4111-9a3e-372d01899e99-kube-api-access-6v529\" (UniqueName: \"kubernetes.io/projected/c08ed4a3-d627-4111-9a3e-372d01899e99-kube-api-access-6v529\") pod \"c08ed4a3-d627-4111-9a3e-372d01899e99\" (UID: \"c08ed4a3-d627-4111-9a3e-372d01899e99\") " Nov 02 23:17:22.265991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.265958 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-openshift-ca" pod "c08ed4a3-d627-4111-9a3e-372d01899e99" (UID: "c08ed4a3-d627-4111-9a3e-372d01899e99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:22.268413 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.268383 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-xks-ca" pod "c08ed4a3-d627-4111-9a3e-372d01899e99" (UID: "c08ed4a3-d627-4111-9a3e-372d01899e99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:17:22.268812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.268795 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08ed4a3-d627-4111-9a3e-372d01899e99-kube-api-access-6v529" pod "c08ed4a3-d627-4111-9a3e-372d01899e99" (UID: "c08ed4a3-d627-4111-9a3e-372d01899e99"). InnerVolumeSpecName "kube-api-access-6v529". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:22.367097 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.367062 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:22.367097 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.367092 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08ed4a3-d627-4111-9a3e-372d01899e99-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:22.367097 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.367104 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v529\" (UniqueName: \"kubernetes.io/projected/c08ed4a3-d627-4111-9a3e-372d01899e99-kube-api-access-6v529\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:22.368612 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.368592 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:22.370621 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.370597 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 23:17:22.370709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.370643 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 23:17:22.911734 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.911699 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" Nov 02 23:17:22.911734 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.911714 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq" event={"ID":"c08ed4a3-d627-4111-9a3e-372d01899e99","Type":"ContainerDied","Data":"9df0c95c888968dea7448bd90fa0e3a818b54e4d8ba837783092634c4151316a"} Nov 02 23:17:22.912310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:22.911804 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9df0c95c888968dea7448bd90fa0e3a818b54e4d8ba837783092634c4151316a" Nov 02 23:17:23.369178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:23.368728 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 23:17:23.369178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:23.368792 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 23:17:24.369388 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:24.369349 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 23:17:24.369852 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:24.369415 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 23:17:25.368686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:25.368653 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 23:17:25.368898 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:25.368706 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 23:17:26.019586 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:17:26.019550 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:17:26.369292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:26.369201 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 23:17:26.369292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:26.369261 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 23:17:27.276445 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:27.276410 5672 scope.go:122] "RemoveContainer" containerID="65241516b3b1cde08cd8997678758b37a7a5ce393d030eefa8061f79250bf4c4" Nov 02 23:17:27.291372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:27.291351 5672 scope.go:122] "RemoveContainer" containerID="cec755b40ea9dd935718eaee65fd7fb18b66e02a188bb13bd23a81f3d63ae558" Nov 02 23:17:27.368930 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:27.368897 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 23:17:27.369084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:27.368967 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 23:17:28.372840 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:28.372809 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:28.373272 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:28.373244 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:28.373945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:28.373922 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:17:32.050585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:32.050545 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r"] Nov 02 23:17:32.099374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:32.099334 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f253b3a-mrr9r"] Nov 02 23:17:32.570273 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:32.570239 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6"] Nov 02 23:17:32.570562 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:32.570526 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" containerID="cri-o://9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b" gracePeriod=30 Nov 02 23:17:32.714209 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:32.714172 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805b59b6-af3e-48bc-97a7-3a62db23ac6e" path="/var/lib/kubelet/pods/805b59b6-af3e-48bc-97a7-3a62db23ac6e/volumes" Nov 02 23:17:33.049007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:33.048962 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2"] Nov 02 23:17:33.058770 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:33.058741 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16e4029-l5sr2"] Nov 02 23:17:34.713463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:34.713432 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16804ae-5327-46aa-82d3-d3df5ab9c2c7" path="/var/lib/kubelet/pods/b16804ae-5327-46aa-82d3-d3df5ab9c2c7/volumes" Nov 02 23:17:37.724831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.718950 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx"] Nov 02 23:17:37.724831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.720390 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c08ed4a3-d627-4111-9a3e-372d01899e99" containerName="revoke-keys" Nov 02 23:17:37.724831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.720425 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08ed4a3-d627-4111-9a3e-372d01899e99" containerName="revoke-keys" Nov 02 23:17:37.724831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.720606 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08ed4a3-d627-4111-9a3e-372d01899e99" containerName="revoke-keys" Nov 02 23:17:37.725504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.724972 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:17:37.787257 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.783607 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx"] Nov 02 23:17:37.823140 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.823105 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-68e067\"" Nov 02 23:17:37.824249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.823522 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-68e067-dockercfg-qknh8\"" Nov 02 23:17:37.838706 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.838620 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd66f\" (UniqueName: \"kubernetes.io/projected/224de738-298d-45a0-849e-389c7f9ee388-kube-api-access-qd66f\") pod \"payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx\" (UID: \"224de738-298d-45a0-849e-389c7f9ee388\") " pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:17:37.839053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.839028 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224de738-298d-45a0-849e-389c7f9ee388-plugins-config-volume\") pod \"payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx\" (UID: \"224de738-298d-45a0-849e-389c7f9ee388\") " pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:17:37.955764 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.954866 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224de738-298d-45a0-849e-389c7f9ee388-plugins-config-volume\") pod \"payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx\" (UID: \"224de738-298d-45a0-849e-389c7f9ee388\") " pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:17:37.955764 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.955077 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd66f\" (UniqueName: \"kubernetes.io/projected/224de738-298d-45a0-849e-389c7f9ee388-kube-api-access-qd66f\") pod \"payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx\" (UID: \"224de738-298d-45a0-849e-389c7f9ee388\") " pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:17:37.956513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.956461 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224de738-298d-45a0-849e-389c7f9ee388-plugins-config-volume\") pod \"payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx\" (UID: \"224de738-298d-45a0-849e-389c7f9ee388\") " pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:17:37.970871 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:37.970850 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:17:38.008671 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.008650 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd66f\" (UniqueName: \"kubernetes.io/projected/224de738-298d-45a0-849e-389c7f9ee388-kube-api-access-qd66f\") pod \"payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx\" (UID: \"224de738-298d-45a0-849e-389c7f9ee388\") " pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:17:38.048621 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.048594 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:17:38.059031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.058084 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-socket\") pod \"db24bfe0-5f82-45c5-b498-4c8b6310177c\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " Nov 02 23:17:38.059031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.058168 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-envoy\") pod \"db24bfe0-5f82-45c5-b498-4c8b6310177c\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " Nov 02 23:17:38.059031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.058222 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-data\") pod \"db24bfe0-5f82-45c5-b498-4c8b6310177c\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " Nov 02 23:17:38.059031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.058269 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-credential-socket\") pod \"db24bfe0-5f82-45c5-b498-4c8b6310177c\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " Nov 02 23:17:38.059031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.058367 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-kube-api-access-m26dn\" (UniqueName: \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-kube-api-access-m26dn\") pod \"db24bfe0-5f82-45c5-b498-4c8b6310177c\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " Nov 02 23:17:38.059031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.058411 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-certs\") pod \"db24bfe0-5f82-45c5-b498-4c8b6310177c\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " Nov 02 23:17:38.059031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.058473 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-token\" (UniqueName: \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-token\") pod \"db24bfe0-5f82-45c5-b498-4c8b6310177c\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " Nov 02 23:17:38.059031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.058536 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-podinfo\") pod \"db24bfe0-5f82-45c5-b498-4c8b6310177c\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " Nov 02 23:17:38.059031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.058670 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db24bfe0-5f82-45c5-b498-4c8b6310177c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db24bfe0-5f82-45c5-b498-4c8b6310177c-istiod-ca-cert\") pod \"db24bfe0-5f82-45c5-b498-4c8b6310177c\" (UID: \"db24bfe0-5f82-45c5-b498-4c8b6310177c\") " Nov 02 23:17:38.059031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.058786 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-credential-socket" pod "db24bfe0-5f82-45c5-b498-4c8b6310177c" (UID: "db24bfe0-5f82-45c5-b498-4c8b6310177c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:38.059603 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.059262 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:38.059603 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.059435 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-socket" pod "db24bfe0-5f82-45c5-b498-4c8b6310177c" (UID: "db24bfe0-5f82-45c5-b498-4c8b6310177c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:38.059603 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.059507 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-data" pod "db24bfe0-5f82-45c5-b498-4c8b6310177c" (UID: "db24bfe0-5f82-45c5-b498-4c8b6310177c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:38.060184 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.059895 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db24bfe0-5f82-45c5-b498-4c8b6310177c-istiod-ca-cert" pod "db24bfe0-5f82-45c5-b498-4c8b6310177c" (UID: "db24bfe0-5f82-45c5-b498-4c8b6310177c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:38.060253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.060220 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-certs" pod "db24bfe0-5f82-45c5-b498-4c8b6310177c" (UID: "db24bfe0-5f82-45c5-b498-4c8b6310177c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:38.062227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.062200 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-kube-api-access-m26dn" pod "db24bfe0-5f82-45c5-b498-4c8b6310177c" (UID: "db24bfe0-5f82-45c5-b498-4c8b6310177c"). InnerVolumeSpecName "kube-api-access-m26dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:38.062540 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.062513 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-podinfo" pod "db24bfe0-5f82-45c5-b498-4c8b6310177c" (UID: "db24bfe0-5f82-45c5-b498-4c8b6310177c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:17:38.062646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.062599 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-envoy" pod "db24bfe0-5f82-45c5-b498-4c8b6310177c" (UID: "db24bfe0-5f82-45c5-b498-4c8b6310177c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:17:38.062717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.062665 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-token" pod "db24bfe0-5f82-45c5-b498-4c8b6310177c" (UID: "db24bfe0-5f82-45c5-b498-4c8b6310177c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:38.091275 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.091193 5672 generic.go:356] "Generic (PLEG): container finished" podID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerID="9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b" exitCode=0 Nov 02 23:17:38.091399 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.091354 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" event={"ID":"db24bfe0-5f82-45c5-b498-4c8b6310177c","Type":"ContainerDied","Data":"9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b"} Nov 02 23:17:38.091469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.091397 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" event={"ID":"db24bfe0-5f82-45c5-b498-4c8b6310177c","Type":"ContainerDied","Data":"efa997097f9654d04c373ddfc41e3e7400b38e6bad890b4294f7b8d2d1a7d4d3"} Nov 02 23:17:38.091469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.091420 5672 scope.go:122] "RemoveContainer" containerID="9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b" Nov 02 23:17:38.091636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.091613 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6" Nov 02 23:17:38.105991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.105971 5672 scope.go:122] "RemoveContainer" containerID="5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3" Nov 02 23:17:38.117044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.117015 5672 scope.go:122] "RemoveContainer" containerID="9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b" Nov 02 23:17:38.117380 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:17:38.117352 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b\": container with ID starting with 9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b not found: ID does not exist" containerID="9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b" Nov 02 23:17:38.117443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.117391 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b"} err="failed to get container status \"9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b\": rpc error: code = NotFound desc = could not find container \"9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b\": container with ID starting with 9079ecc9e529719e9ded833f609c260c79b681720ff91418efb785d73358000b not found: ID does not exist" Nov 02 23:17:38.117443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.117418 5672 scope.go:122] "RemoveContainer" containerID="5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3" Nov 02 23:17:38.117669 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:17:38.117646 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3\": container with ID starting with 5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3 not found: ID does not exist" containerID="5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3" Nov 02 23:17:38.117720 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.117681 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3"} err="failed to get container status \"5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3\": rpc error: code = NotFound desc = could not find container \"5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3\": container with ID starting with 5d87b0e0d71b7f26d351a06d44254dbe625014b32b028f5a59b1380d27e1d7a3 not found: ID does not exist" Nov 02 23:17:38.160540 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.160399 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m26dn\" (UniqueName: \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-kube-api-access-m26dn\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:38.160540 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.160433 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:38.160540 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.160449 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:38.160540 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.160463 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:38.160540 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.160476 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db24bfe0-5f82-45c5-b498-4c8b6310177c-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:38.160540 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.160490 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:38.160540 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.160504 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:38.160540 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.160518 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db24bfe0-5f82-45c5-b498-4c8b6310177c-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:38.249071 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.249038 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6"] Nov 02 23:17:38.252939 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.252909 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b2a689-openshift-default-6745676dfc-zjrc6"] Nov 02 23:17:38.447659 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:17:38.447625 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224de738_298d_45a0_849e_389c7f9ee388.slice/crio-5c5396d117b085f1379a25ddbe5246fcb0b0c88bcf52defa0825b8b984cff701 WatchSource:0}: Error finding container 5c5396d117b085f1379a25ddbe5246fcb0b0c88bcf52defa0825b8b984cff701: Status 404 returned error can't find the container with id 5c5396d117b085f1379a25ddbe5246fcb0b0c88bcf52defa0825b8b984cff701 Nov 02 23:17:38.448614 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.448584 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx"] Nov 02 23:17:38.713366 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:38.713291 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" path="/var/lib/kubelet/pods/db24bfe0-5f82-45c5-b498-4c8b6310177c/volumes" Nov 02 23:17:39.100355 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:39.100319 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" event={"ID":"224de738-298d-45a0-849e-389c7f9ee388","Type":"ContainerStarted","Data":"cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364"} Nov 02 23:17:39.100805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:39.100373 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" event={"ID":"224de738-298d-45a0-849e-389c7f9ee388","Type":"ContainerStarted","Data":"5c5396d117b085f1379a25ddbe5246fcb0b0c88bcf52defa0825b8b984cff701"} Nov 02 23:17:39.100805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:39.100397 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:17:39.209793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:39.209744 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" podStartSLOduration=2.209730103 podStartE2EDuration="2.209730103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:39.20654308 +0000 UTC m=+1452.613854909" watchObservedRunningTime="2026-11-02 23:17:39.209730103 +0000 UTC m=+1452.617041946" Nov 02 23:17:40.044514 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:40.044470 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g"] Nov 02 23:17:40.090884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:40.090844 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5482e69-p2g6g"] Nov 02 23:17:40.417669 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:40.417572 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn"] Nov 02 23:17:40.518447 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:40.518410 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x"] Nov 02 23:17:40.716991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:40.716905 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9755d8-1445-4e6c-bd03-d69af2280d17" path="/var/lib/kubelet/pods/8b9755d8-1445-4e6c-bd03-d69af2280d17/volumes" Nov 02 23:17:41.043431 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:41.043399 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp"] Nov 02 23:17:41.047104 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:41.047072 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fcf7ecf-w6qtp"] Nov 02 23:17:42.011470 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:42.011437 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:17:42.714065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:42.714027 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d7bab5-706c-474b-a725-77fb942984ad" path="/var/lib/kubelet/pods/77d7bab5-706c-474b-a725-77fb942984ad/volumes" Nov 02 23:17:45.418125 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.418059 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" podUID="03781ef2-e363-4467-afce-09dc3a4ab0bf" containerName="payload-processing" containerID="cri-o://c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75" gracePeriod=25 Nov 02 23:17:45.518729 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.518689 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" podUID="2370ad5b-63c5-4296-9f33-ff056ea691ae" containerName="payload-pre-processing" containerID="cri-o://deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993" gracePeriod=25 Nov 02 23:17:45.693893 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.693866 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:17:45.774881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.774840 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03781ef2-e363-4467-afce-09dc3a4ab0bf-kube-api-access-9g4cs\" (UniqueName: \"kubernetes.io/projected/03781ef2-e363-4467-afce-09dc3a4ab0bf-kube-api-access-9g4cs\") pod \"03781ef2-e363-4467-afce-09dc3a4ab0bf\" (UID: \"03781ef2-e363-4467-afce-09dc3a4ab0bf\") " Nov 02 23:17:45.775067 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.774981 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03781ef2-e363-4467-afce-09dc3a4ab0bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03781ef2-e363-4467-afce-09dc3a4ab0bf-plugins-config-volume\") pod \"03781ef2-e363-4467-afce-09dc3a4ab0bf\" (UID: \"03781ef2-e363-4467-afce-09dc3a4ab0bf\") " Nov 02 23:17:45.776924 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.776893 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03781ef2-e363-4467-afce-09dc3a4ab0bf-plugins-config-volume" pod "03781ef2-e363-4467-afce-09dc3a4ab0bf" (UID: "03781ef2-e363-4467-afce-09dc3a4ab0bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:45.778466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.778439 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03781ef2-e363-4467-afce-09dc3a4ab0bf-kube-api-access-9g4cs" pod "03781ef2-e363-4467-afce-09dc3a4ab0bf" (UID: "03781ef2-e363-4467-afce-09dc3a4ab0bf"). InnerVolumeSpecName "kube-api-access-9g4cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:45.799676 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.799649 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:17:45.877014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.876984 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2370ad5b-63c5-4296-9f33-ff056ea691ae-kube-api-access-qvc24\" (UniqueName: \"kubernetes.io/projected/2370ad5b-63c5-4296-9f33-ff056ea691ae-kube-api-access-qvc24\") pod \"2370ad5b-63c5-4296-9f33-ff056ea691ae\" (UID: \"2370ad5b-63c5-4296-9f33-ff056ea691ae\") " Nov 02 23:17:45.877260 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.877075 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2370ad5b-63c5-4296-9f33-ff056ea691ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2370ad5b-63c5-4296-9f33-ff056ea691ae-plugins-config-volume\") pod \"2370ad5b-63c5-4296-9f33-ff056ea691ae\" (UID: \"2370ad5b-63c5-4296-9f33-ff056ea691ae\") " Nov 02 23:17:45.877500 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.877471 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2370ad5b-63c5-4296-9f33-ff056ea691ae-plugins-config-volume" pod "2370ad5b-63c5-4296-9f33-ff056ea691ae" (UID: "2370ad5b-63c5-4296-9f33-ff056ea691ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:17:45.877746 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.877724 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g4cs\" (UniqueName: \"kubernetes.io/projected/03781ef2-e363-4467-afce-09dc3a4ab0bf-kube-api-access-9g4cs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:45.877805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.877752 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03781ef2-e363-4467-afce-09dc3a4ab0bf-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:45.877805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.877768 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2370ad5b-63c5-4296-9f33-ff056ea691ae-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:45.879341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.879316 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2370ad5b-63c5-4296-9f33-ff056ea691ae-kube-api-access-qvc24" pod "2370ad5b-63c5-4296-9f33-ff056ea691ae" (UID: "2370ad5b-63c5-4296-9f33-ff056ea691ae"). InnerVolumeSpecName "kube-api-access-qvc24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:17:45.978917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:45.978883 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvc24\" (UniqueName: \"kubernetes.io/projected/2370ad5b-63c5-4296-9f33-ff056ea691ae-kube-api-access-qvc24\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:17:46.170989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.170955 5672 generic.go:356] "Generic (PLEG): container finished" podID="03781ef2-e363-4467-afce-09dc3a4ab0bf" containerID="c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75" exitCode=0 Nov 02 23:17:46.171210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.171011 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" Nov 02 23:17:46.171210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.171074 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" event={"ID":"03781ef2-e363-4467-afce-09dc3a4ab0bf","Type":"ContainerDied","Data":"c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75"} Nov 02 23:17:46.171210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.171121 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn" event={"ID":"03781ef2-e363-4467-afce-09dc3a4ab0bf","Type":"ContainerDied","Data":"7656bd4f613eb77b3c49ceedae897d80b36cee6138926a323318fb5e0f7a9941"} Nov 02 23:17:46.171210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.171164 5672 scope.go:122] "RemoveContainer" containerID="c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75" Nov 02 23:17:46.173282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.172989 5672 generic.go:356] "Generic (PLEG): container finished" podID="2370ad5b-63c5-4296-9f33-ff056ea691ae" containerID="deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993" exitCode=0 Nov 02 23:17:46.173282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.173065 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" event={"ID":"2370ad5b-63c5-4296-9f33-ff056ea691ae","Type":"ContainerDied","Data":"deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993"} Nov 02 23:17:46.173282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.173118 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" event={"ID":"2370ad5b-63c5-4296-9f33-ff056ea691ae","Type":"ContainerDied","Data":"8f7d081a03edd2ae0dc5bac3c95790a971244e85e7cbab28f83c89170fc2b399"} Nov 02 23:17:46.173282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.173200 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x" Nov 02 23:17:46.183257 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.183238 5672 scope.go:122] "RemoveContainer" containerID="c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75" Nov 02 23:17:46.183763 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:17:46.183739 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75\": container with ID starting with c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75 not found: ID does not exist" containerID="c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75" Nov 02 23:17:46.183875 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.183774 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75"} err="failed to get container status \"c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75\": rpc error: code = NotFound desc = could not find container \"c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75\": container with ID starting with c775062ce14c0f15c6111fa083c9b4832ccb380660d9fb19fc0a1d644e641c75 not found: ID does not exist" Nov 02 23:17:46.183875 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.183797 5672 scope.go:122] "RemoveContainer" containerID="deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993" Nov 02 23:17:46.193633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.193616 5672 scope.go:122] "RemoveContainer" containerID="deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993" Nov 02 23:17:46.193889 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:17:46.193871 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993\": container with ID starting with deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993 not found: ID does not exist" containerID="deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993" Nov 02 23:17:46.193931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.193898 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993"} err="failed to get container status \"deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993\": rpc error: code = NotFound desc = could not find container \"deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993\": container with ID starting with deaead5a69bf7b7a315f80025c3a66ad60111e5677af544c07e08a015e55c993 not found: ID does not exist" Nov 02 23:17:46.326488 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.326407 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x"] Nov 02 23:17:46.331061 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.331025 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f46208-86bb775895-t7p8x"] Nov 02 23:17:46.431380 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.431343 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn"] Nov 02 23:17:46.435335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.435307 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f46208-6fc88577c7-glwzn"] Nov 02 23:17:46.713932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.713854 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03781ef2-e363-4467-afce-09dc3a4ab0bf" path="/var/lib/kubelet/pods/03781ef2-e363-4467-afce-09dc3a4ab0bf/volumes" Nov 02 23:17:46.714258 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:46.714244 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2370ad5b-63c5-4296-9f33-ff056ea691ae" path="/var/lib/kubelet/pods/2370ad5b-63c5-4296-9f33-ff056ea691ae/volumes" Nov 02 23:17:50.044604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:50.044571 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw"] Nov 02 23:17:50.091597 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:50.091562 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb"] Nov 02 23:17:50.095076 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:50.095050 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6219c92-h7zfw"] Nov 02 23:17:50.098335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:50.098306 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75cdb93f-rmrrb"] Nov 02 23:17:50.111297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:50.111275 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:17:50.714245 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:50.714212 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b71bea1-31a1-4e7b-b913-8f305a3af254" path="/var/lib/kubelet/pods/9b71bea1-31a1-4e7b-b913-8f305a3af254/volumes" Nov 02 23:17:50.714719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:50.714703 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a1a57c-d16d-4709-b751-53112ed20714" path="/var/lib/kubelet/pods/f6a1a57c-d16d-4709-b751-53112ed20714/volumes" Nov 02 23:17:55.042969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.042928 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w"] Nov 02 23:17:55.043784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.043763 5672 cpu_manager.go:414] "containerMap: removing container" podUID="03781ef2-e363-4467-afce-09dc3a4ab0bf" containerName="payload-processing" Nov 02 23:17:55.043829 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.043788 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03781ef2-e363-4467-afce-09dc3a4ab0bf" containerName="payload-processing" Nov 02 23:17:55.043863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.043835 5672 cpu_manager.go:414] "containerMap: removing container" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" Nov 02 23:17:55.043863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.043843 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" Nov 02 23:17:55.043863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.043855 5672 cpu_manager.go:414] "containerMap: removing container" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" Nov 02 23:17:55.043956 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.043864 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" Nov 02 23:17:55.043956 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.043887 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2370ad5b-63c5-4296-9f33-ff056ea691ae" containerName="payload-pre-processing" Nov 02 23:17:55.043956 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.043894 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2370ad5b-63c5-4296-9f33-ff056ea691ae" containerName="payload-pre-processing" Nov 02 23:17:55.044079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.044065 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="03781ef2-e363-4467-afce-09dc3a4ab0bf" containerName="payload-processing" Nov 02 23:17:55.044115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.044089 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" Nov 02 23:17:55.044115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.044101 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="db24bfe0-5f82-45c5-b498-4c8b6310177c" containerName="istio-proxy" Nov 02 23:17:55.044115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.044111 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2370ad5b-63c5-4296-9f33-ff056ea691ae" containerName="payload-pre-processing" Nov 02 23:17:55.049652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.049629 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.106203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.100343 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-79385a-openshift-default-dockercfg-qq5bp\"" Nov 02 23:17:55.114222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.111947 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-credential-socket\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.114222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.112042 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-socket\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.114222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.112100 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-data\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.114222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.112343 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-envoy\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.114222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.112401 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-certs\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.114222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.112432 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2dccd-0e58-471e-92c4-9981abf92255-istiod-ca-cert\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.114222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.112460 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-podinfo\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.114222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.112536 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9rds\" (UniqueName: \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-kube-api-access-q9rds\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.114222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.112584 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-token\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.168646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.161253 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w"] Nov 02 23:17:55.214296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.213773 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-socket\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.214296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.213830 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-data\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.214296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.213895 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-envoy\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.214296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.213926 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-certs\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.214296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.213942 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2dccd-0e58-471e-92c4-9981abf92255-istiod-ca-cert\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.214296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.213965 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-podinfo\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.214296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.214037 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9rds\" (UniqueName: \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-kube-api-access-q9rds\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.214296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.214102 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-token\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.214855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.214632 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-certs\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.215427 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.215191 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-credential-socket\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.215717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.215691 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-data\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.216296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.215915 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-socket\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.216296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.215951 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-credential-socket\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.223526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.216923 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2dccd-0e58-471e-92c4-9981abf92255-istiod-ca-cert\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.224424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.224396 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-podinfo\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.225970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.225951 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-envoy\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.269786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.269750 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-token\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.270225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.270205 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9rds\" (UniqueName: \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-kube-api-access-q9rds\") pod \"e2e-api-b-79385a-openshift-default-8645744ccc-89b7w\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.395199 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.395097 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:55.842090 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.841793 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w"] Nov 02 23:17:55.843071 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:17:55.843046 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc2dccd_0e58_471e_92c4_9981abf92255.slice/crio-266a94578ffeb709fc4b66760ff0a51523f05e09aca9bbf0d891eee4f122588f WatchSource:0}: Error finding container 266a94578ffeb709fc4b66760ff0a51523f05e09aca9bbf0d891eee4f122588f: Status 404 returned error can't find the container with id 266a94578ffeb709fc4b66760ff0a51523f05e09aca9bbf0d891eee4f122588f Nov 02 23:17:55.845819 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.845789 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:55.845917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.845900 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:55.845985 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:55.845949 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:17:56.024749 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:17:56.024700 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:17:56.261052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:56.261013 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" event={"ID":"ccc2dccd-0e58-471e-92c4-9981abf92255","Type":"ContainerStarted","Data":"a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa"} Nov 02 23:17:56.261565 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:56.261083 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" event={"ID":"ccc2dccd-0e58-471e-92c4-9981abf92255","Type":"ContainerStarted","Data":"266a94578ffeb709fc4b66760ff0a51523f05e09aca9bbf0d891eee4f122588f"} Nov 02 23:17:56.374053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:56.373996 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" podStartSLOduration=2.3739814470000002 podStartE2EDuration="2.373981447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:56.37021151 +0000 UTC m=+1469.777523343" watchObservedRunningTime="2026-11-02 23:17:56.373981447 +0000 UTC m=+1469.781293281" Nov 02 23:17:56.395406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:56.395374 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:56.400417 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:56.400393 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:57.269193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:57.269159 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:17:57.270069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:17:57.270048 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:18:01.044859 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:01.044823 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp"] Nov 02 23:18:01.076725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:01.076694 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7403b508-shqxp"] Nov 02 23:18:02.704418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.704380 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44"] Nov 02 23:18:02.710328 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.710287 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:18:02.724943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.724910 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32c17e66-5390-4350-acc8-007d6a6192ea" path="/var/lib/kubelet/pods/32c17e66-5390-4350-acc8-007d6a6192ea/volumes" Nov 02 23:18:02.768490 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.768438 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44"] Nov 02 23:18:02.804991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.804958 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-79385a-dockercfg-rfwmb\"" Nov 02 23:18:02.804991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.804965 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-79385a\"" Nov 02 23:18:02.824426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.824393 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd54c36b-d57e-46dc-953d-e6b446d17007-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44\" (UID: \"bd54c36b-d57e-46dc-953d-e6b446d17007\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:18:02.824603 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.824543 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2b2n\" (UniqueName: \"kubernetes.io/projected/bd54c36b-d57e-46dc-953d-e6b446d17007-kube-api-access-m2b2n\") pod \"payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44\" (UID: \"bd54c36b-d57e-46dc-953d-e6b446d17007\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:18:02.865870 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.865827 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs"] Nov 02 23:18:02.871388 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.871363 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:18:02.931989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.931927 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2b2n\" (UniqueName: \"kubernetes.io/projected/bd54c36b-d57e-46dc-953d-e6b446d17007-kube-api-access-m2b2n\") pod \"payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44\" (UID: \"bd54c36b-d57e-46dc-953d-e6b446d17007\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:18:02.932545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.932507 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd54c36b-d57e-46dc-953d-e6b446d17007-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44\" (UID: \"bd54c36b-d57e-46dc-953d-e6b446d17007\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:18:02.933671 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.933642 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd54c36b-d57e-46dc-953d-e6b446d17007-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44\" (UID: \"bd54c36b-d57e-46dc-953d-e6b446d17007\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:18:02.951433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.946626 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs"] Nov 02 23:18:02.997207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:02.990274 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2b2n\" (UniqueName: \"kubernetes.io/projected/bd54c36b-d57e-46dc-953d-e6b446d17007-kube-api-access-m2b2n\") pod \"payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44\" (UID: \"bd54c36b-d57e-46dc-953d-e6b446d17007\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:18:03.032655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:03.032621 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:18:03.034232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:03.034198 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8n8d\" (UniqueName: \"kubernetes.io/projected/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-kube-api-access-p8n8d\") pod \"payload-processing-e2e-api-b-79385a-c75d5f567-hphqs\" (UID: \"39b34d72-b9df-4e1e-a0c1-a5d620060bfb\") " pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:18:03.034385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:03.034248 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-79385a-c75d5f567-hphqs\" (UID: \"39b34d72-b9df-4e1e-a0c1-a5d620060bfb\") " pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:18:03.136540 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:03.136507 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8n8d\" (UniqueName: \"kubernetes.io/projected/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-kube-api-access-p8n8d\") pod \"payload-processing-e2e-api-b-79385a-c75d5f567-hphqs\" (UID: \"39b34d72-b9df-4e1e-a0c1-a5d620060bfb\") " pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:18:03.136658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:03.136555 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-79385a-c75d5f567-hphqs\" (UID: \"39b34d72-b9df-4e1e-a0c1-a5d620060bfb\") " pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:18:03.137537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:03.137513 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-79385a-c75d5f567-hphqs\" (UID: \"39b34d72-b9df-4e1e-a0c1-a5d620060bfb\") " pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:18:03.193951 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:03.193875 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8n8d\" (UniqueName: \"kubernetes.io/projected/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-kube-api-access-p8n8d\") pod \"payload-processing-e2e-api-b-79385a-c75d5f567-hphqs\" (UID: \"39b34d72-b9df-4e1e-a0c1-a5d620060bfb\") " pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:18:03.442261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:03.442230 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44"] Nov 02 23:18:03.444224 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:18:03.444197 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd54c36b_d57e_46dc_953d_e6b446d17007.slice/crio-c7451632679e3953b73841523261a14e1612e35506ea54d1a144f1629d625702 WatchSource:0}: Error finding container c7451632679e3953b73841523261a14e1612e35506ea54d1a144f1629d625702: Status 404 returned error can't find the container with id c7451632679e3953b73841523261a14e1612e35506ea54d1a144f1629d625702 Nov 02 23:18:03.491718 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:03.491693 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:18:03.869703 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:18:03.869671 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b34d72_b9df_4e1e_a0c1_a5d620060bfb.slice/crio-804a8fcc0364815ce2b91ab6b2daed96c69b31e07c42f552908e59ec8cf16cc8 WatchSource:0}: Error finding container 804a8fcc0364815ce2b91ab6b2daed96c69b31e07c42f552908e59ec8cf16cc8: Status 404 returned error can't find the container with id 804a8fcc0364815ce2b91ab6b2daed96c69b31e07c42f552908e59ec8cf16cc8 Nov 02 23:18:03.870087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:03.869842 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs"] Nov 02 23:18:04.331553 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:04.331523 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" event={"ID":"39b34d72-b9df-4e1e-a0c1-a5d620060bfb","Type":"ContainerStarted","Data":"5e2c5684025a5bb1a58104176133aa22c64400680afcfa3076b261e3f19ec36a"} Nov 02 23:18:04.331755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:04.331577 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" event={"ID":"39b34d72-b9df-4e1e-a0c1-a5d620060bfb","Type":"ContainerStarted","Data":"804a8fcc0364815ce2b91ab6b2daed96c69b31e07c42f552908e59ec8cf16cc8"} Nov 02 23:18:04.331755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:04.331611 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:18:04.333224 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:04.333195 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" event={"ID":"bd54c36b-d57e-46dc-953d-e6b446d17007","Type":"ContainerStarted","Data":"376c4a12d3d362614152cf58b16a25987d72b69068a19d57f87e6901821967a7"} Nov 02 23:18:04.333383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:04.333240 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" event={"ID":"bd54c36b-d57e-46dc-953d-e6b446d17007","Type":"ContainerStarted","Data":"c7451632679e3953b73841523261a14e1612e35506ea54d1a144f1629d625702"} Nov 02 23:18:04.333383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:04.333305 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:18:04.437512 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:04.437463 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" podStartSLOduration=2.437448579 podStartE2EDuration="2.437448579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:04.436942714 +0000 UTC m=+1477.844254545" watchObservedRunningTime="2026-11-02 23:18:04.437448579 +0000 UTC m=+1477.844760414" Nov 02 23:18:04.549218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:04.549121 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" podStartSLOduration=2.549106709 podStartE2EDuration="2.549106709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:04.543567388 +0000 UTC m=+1477.950879218" watchObservedRunningTime="2026-11-02 23:18:04.549106709 +0000 UTC m=+1477.956418539" Nov 02 23:18:05.046943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:05.046914 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g"] Nov 02 23:18:05.050198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:05.050171 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5066a43-ngl2g"] Nov 02 23:18:06.734973 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:06.734934 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c972f72-b716-434a-a2fa-49cfcd77aac3" path="/var/lib/kubelet/pods/9c972f72-b716-434a-a2fa-49cfcd77aac3/volumes" Nov 02 23:18:09.032302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:09.032269 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h"] Nov 02 23:18:09.077905 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:09.077875 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-xct7h"] Nov 02 23:18:10.713336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:10.713303 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50d8153e-748b-4cbc-bbbd-61d4f0e08c41" path="/var/lib/kubelet/pods/50d8153e-748b-4cbc-bbbd-61d4f0e08c41/volumes" Nov 02 23:18:12.047184 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:12.047137 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj"] Nov 02 23:18:12.093449 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:12.093414 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ed25c8-84gpj"] Nov 02 23:18:12.713740 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:12.713703 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b26163c-1466-453b-9685-eaace6d1bfbe" path="/var/lib/kubelet/pods/3b26163c-1466-453b-9685-eaace6d1bfbe/volumes" Nov 02 23:18:15.176816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:15.176777 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt"] Nov 02 23:18:15.177311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:15.177016 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" containerID="cri-o://77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0" gracePeriod=30 Nov 02 23:18:15.356622 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:15.356592 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:18:15.361648 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:15.361619 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:18:17.043967 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:17.043931 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j"] Nov 02 23:18:17.090070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:17.090035 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-217eb132-mhg7j"] Nov 02 23:18:18.717064 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:18.717024 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d6ee286-bb1c-42d5-a5ea-800f8de9d05f" path="/var/lib/kubelet/pods/3d6ee286-bb1c-42d5-a5ea-800f8de9d05f/volumes" Nov 02 23:18:18.879980 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:18.879942 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:18:19.321640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:19.321604 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:18:19.321818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:19.321669 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:18:20.458397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.458331 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:18:20.472975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.472948 5672 generic.go:356] "Generic (PLEG): container finished" podID="09a7e949-3e53-406b-83ec-9680954ad105" containerID="77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0" exitCode=0 Nov 02 23:18:20.473112 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.472992 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" Nov 02 23:18:20.473112 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.473033 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" event={"ID":"09a7e949-3e53-406b-83ec-9680954ad105","Type":"ContainerDied","Data":"77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0"} Nov 02 23:18:20.473112 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.473089 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt" event={"ID":"09a7e949-3e53-406b-83ec-9680954ad105","Type":"ContainerDied","Data":"740b749d3bbd188805237cdd2984cb1bff434179adb86269db090d69f4f37cd5"} Nov 02 23:18:20.473255 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.473113 5672 scope.go:122] "RemoveContainer" containerID="77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0" Nov 02 23:18:20.491198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.491173 5672 scope.go:122] "RemoveContainer" containerID="77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0" Nov 02 23:18:20.491547 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:18:20.491521 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0\": container with ID starting with 77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0 not found: ID does not exist" containerID="77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0" Nov 02 23:18:20.491609 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.491552 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0"} err="failed to get container status \"77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0\": rpc error: code = NotFound desc = could not find container \"77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0\": container with ID starting with 77a345256a9ad3be026c4e807ecff1720bbec62dffcaa2802f4f815e6fef1ac0 not found: ID does not exist" Nov 02 23:18:20.516117 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516086 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a7e949-3e53-406b-83ec-9680954ad105-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a7e949-3e53-406b-83ec-9680954ad105-istiod-ca-cert\") pod \"09a7e949-3e53-406b-83ec-9680954ad105\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " Nov 02 23:18:20.516289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516183 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-data\") pod \"09a7e949-3e53-406b-83ec-9680954ad105\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " Nov 02 23:18:20.516289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516228 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-certs\") pod \"09a7e949-3e53-406b-83ec-9680954ad105\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " Nov 02 23:18:20.516289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516264 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-credential-socket\") pod \"09a7e949-3e53-406b-83ec-9680954ad105\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " Nov 02 23:18:20.516457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516333 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-istio-token\" (UniqueName: \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-istio-token\") pod \"09a7e949-3e53-406b-83ec-9680954ad105\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " Nov 02 23:18:20.516457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516405 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09a7e949-3e53-406b-83ec-9680954ad105-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a7e949-3e53-406b-83ec-9680954ad105-istio-podinfo\") pod \"09a7e949-3e53-406b-83ec-9680954ad105\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " Nov 02 23:18:20.516562 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516467 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-socket\") pod \"09a7e949-3e53-406b-83ec-9680954ad105\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " Nov 02 23:18:20.516562 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516512 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-envoy\") pod \"09a7e949-3e53-406b-83ec-9680954ad105\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " Nov 02 23:18:20.516662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516554 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a7e949-3e53-406b-83ec-9680954ad105-istiod-ca-cert" pod "09a7e949-3e53-406b-83ec-9680954ad105" (UID: "09a7e949-3e53-406b-83ec-9680954ad105"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:20.516662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516556 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-data" pod "09a7e949-3e53-406b-83ec-9680954ad105" (UID: "09a7e949-3e53-406b-83ec-9680954ad105"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.516662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516574 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-certs" pod "09a7e949-3e53-406b-83ec-9680954ad105" (UID: "09a7e949-3e53-406b-83ec-9680954ad105"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.516662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516644 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-credential-socket" pod "09a7e949-3e53-406b-83ec-9680954ad105" (UID: "09a7e949-3e53-406b-83ec-9680954ad105"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.516872 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516821 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-kube-api-access-9s277\" (UniqueName: \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-kube-api-access-9s277\") pod \"09a7e949-3e53-406b-83ec-9680954ad105\" (UID: \"09a7e949-3e53-406b-83ec-9680954ad105\") " Nov 02 23:18:20.516872 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.516836 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-socket" pod "09a7e949-3e53-406b-83ec-9680954ad105" (UID: "09a7e949-3e53-406b-83ec-9680954ad105"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.517799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.517773 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.517931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.517809 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a7e949-3e53-406b-83ec-9680954ad105-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.517931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.517828 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.517931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.517843 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.517931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.517864 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.519299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.519269 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-envoy" pod "09a7e949-3e53-406b-83ec-9680954ad105" (UID: "09a7e949-3e53-406b-83ec-9680954ad105"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:20.519428 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.519335 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-kube-api-access-9s277" pod "09a7e949-3e53-406b-83ec-9680954ad105" (UID: "09a7e949-3e53-406b-83ec-9680954ad105"). InnerVolumeSpecName "kube-api-access-9s277". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:20.522321 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.522276 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-istio-token" pod "09a7e949-3e53-406b-83ec-9680954ad105" (UID: "09a7e949-3e53-406b-83ec-9680954ad105"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:20.522410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.522322 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09a7e949-3e53-406b-83ec-9680954ad105-istio-podinfo" pod "09a7e949-3e53-406b-83ec-9680954ad105" (UID: "09a7e949-3e53-406b-83ec-9680954ad105"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:20.619495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.619462 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s277\" (UniqueName: \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-kube-api-access-9s277\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.619495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.619497 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a7e949-3e53-406b-83ec-9680954ad105-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.619680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.619514 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a7e949-3e53-406b-83ec-9680954ad105-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.619680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.619527 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a7e949-3e53-406b-83ec-9680954ad105-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:20.898072 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.898036 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt"] Nov 02 23:18:20.903900 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:20.903872 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-w95gt"] Nov 02 23:18:22.712692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:22.712658 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a7e949-3e53-406b-83ec-9680954ad105" path="/var/lib/kubelet/pods/09a7e949-3e53-406b-83ec-9680954ad105/volumes" Nov 02 23:18:26.021094 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:18:26.021050 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:18:27.349982 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:27.349947 5672 scope.go:122] "RemoveContainer" containerID="103000fa8c498c2bdc3e21150c77ba47f0dac6fe33882bc83c7864315477bf98" Nov 02 23:18:27.362772 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:27.362751 5672 scope.go:122] "RemoveContainer" containerID="77352e0ac2ace52d3e61a93916caacbb948a7f7f0a91e9735c19ef89ca0525e1" Nov 02 23:18:27.373953 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:27.373932 5672 scope.go:122] "RemoveContainer" containerID="756da09516285c7f5711e0b08325eda4c25b4be3ff662f91a4fc92d26f4b1d91" Nov 02 23:18:27.382599 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:27.382578 5672 scope.go:122] "RemoveContainer" containerID="33b4254f27728124cb73d44c0fad01d76b7d42696493f77ac79411e52480370c" Nov 02 23:18:27.390987 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:27.390969 5672 scope.go:122] "RemoveContainer" containerID="0fdc5a43743355daed110e04dd6b1a6e945ee5e3716fe5500c280421b038c7b8" Nov 02 23:18:27.399405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:27.399387 5672 scope.go:122] "RemoveContainer" containerID="98704f72d8fa163d6a5aa442186e940899902b43c015c030470c42c4a0b96e60" Nov 02 23:18:37.980536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:37.980485 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9"] Nov 02 23:18:37.981251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:37.981230 5672 cpu_manager.go:414] "containerMap: removing container" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" Nov 02 23:18:37.981251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:37.981251 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" Nov 02 23:18:37.981465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:37.981452 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a7e949-3e53-406b-83ec-9680954ad105" containerName="istio-proxy" Nov 02 23:18:37.987232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:37.987211 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:38.036412 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.036384 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:18:38.039350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.039321 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9"] Nov 02 23:18:38.081334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.081306 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:18:38.081529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.081324 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:18:38.081529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.081424 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:18:38.141249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.141216 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff0eef6-mcbx9\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:38.141413 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.141346 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff0eef6-mcbx9\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:38.141413 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.141384 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjrz\" (UniqueName: \"kubernetes.io/projected/cdfd4a95-9541-430b-baee-364f8c1b5da1-kube-api-access-fvjrz\") pod \"maas-api-revoke-sub-cff0eef6-mcbx9\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:38.242655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.242577 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff0eef6-mcbx9\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:38.242786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.242678 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff0eef6-mcbx9\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:38.242786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.242710 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjrz\" (UniqueName: \"kubernetes.io/projected/cdfd4a95-9541-430b-baee-364f8c1b5da1-kube-api-access-fvjrz\") pod \"maas-api-revoke-sub-cff0eef6-mcbx9\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:38.243409 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.243376 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff0eef6-mcbx9\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:38.244896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.244875 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff0eef6-mcbx9\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:38.295949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.295914 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjrz\" (UniqueName: \"kubernetes.io/projected/cdfd4a95-9541-430b-baee-364f8c1b5da1-kube-api-access-fvjrz\") pod \"maas-api-revoke-sub-cff0eef6-mcbx9\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:38.300520 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.300492 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:38.676195 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:38.676163 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9"] Nov 02 23:18:38.676880 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:18:38.676855 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfd4a95_9541_430b_baee_364f8c1b5da1.slice/crio-09a56eec721bef0a0ba0fca6e0c63c2b6f381ded1942568cbb72997492aa4bb0 WatchSource:0}: Error finding container 09a56eec721bef0a0ba0fca6e0c63c2b6f381ded1942568cbb72997492aa4bb0: Status 404 returned error can't find the container with id 09a56eec721bef0a0ba0fca6e0c63c2b6f381ded1942568cbb72997492aa4bb0 Nov 02 23:18:39.619533 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:39.619492 5672 generic.go:356] "Generic (PLEG): container finished" podID="cdfd4a95-9541-430b-baee-364f8c1b5da1" containerID="cd00a841b8c80b904db717e2bcd70473da5fec3c490f77db6060420e95b29f79" exitCode=0 Nov 02 23:18:39.619928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:39.619576 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" event={"ID":"cdfd4a95-9541-430b-baee-364f8c1b5da1","Type":"ContainerDied","Data":"cd00a841b8c80b904db717e2bcd70473da5fec3c490f77db6060420e95b29f79"} Nov 02 23:18:39.619928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:39.619675 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" event={"ID":"cdfd4a95-9541-430b-baee-364f8c1b5da1","Type":"ContainerStarted","Data":"09a56eec721bef0a0ba0fca6e0c63c2b6f381ded1942568cbb72997492aa4bb0"} Nov 02 23:18:40.044264 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:40.044235 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72"] Nov 02 23:18:40.047448 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:40.047420 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb4e60f-8xx72"] Nov 02 23:18:40.714604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:40.714568 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d50736-194e-42c4-b972-d0407bb2fe6d" path="/var/lib/kubelet/pods/d7d50736-194e-42c4-b972-d0407bb2fe6d/volumes" Nov 02 23:18:40.784999 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:40.784973 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:40.985408 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:40.985373 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-xks-ca\") pod \"cdfd4a95-9541-430b-baee-364f8c1b5da1\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " Nov 02 23:18:40.985573 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:40.985417 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-openshift-ca\") pod \"cdfd4a95-9541-430b-baee-364f8c1b5da1\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " Nov 02 23:18:40.985573 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:40.985483 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdfd4a95-9541-430b-baee-364f8c1b5da1-kube-api-access-fvjrz\" (UniqueName: \"kubernetes.io/projected/cdfd4a95-9541-430b-baee-364f8c1b5da1-kube-api-access-fvjrz\") pod \"cdfd4a95-9541-430b-baee-364f8c1b5da1\" (UID: \"cdfd4a95-9541-430b-baee-364f8c1b5da1\") " Nov 02 23:18:40.985842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:40.985818 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-openshift-ca" pod "cdfd4a95-9541-430b-baee-364f8c1b5da1" (UID: "cdfd4a95-9541-430b-baee-364f8c1b5da1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:40.985973 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:40.985959 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:40.987488 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:40.987466 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdfd4a95-9541-430b-baee-364f8c1b5da1-kube-api-access-fvjrz" pod "cdfd4a95-9541-430b-baee-364f8c1b5da1" (UID: "cdfd4a95-9541-430b-baee-364f8c1b5da1"). InnerVolumeSpecName "kube-api-access-fvjrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:40.987578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:40.987532 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-xks-ca" pod "cdfd4a95-9541-430b-baee-364f8c1b5da1" (UID: "cdfd4a95-9541-430b-baee-364f8c1b5da1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:18:41.086887 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:41.086860 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvjrz\" (UniqueName: \"kubernetes.io/projected/cdfd4a95-9541-430b-baee-364f8c1b5da1-kube-api-access-fvjrz\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:41.086887 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:41.086885 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfd4a95-9541-430b-baee-364f8c1b5da1-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:41.636044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:41.636014 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" Nov 02 23:18:41.636334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:41.636039 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9" event={"ID":"cdfd4a95-9541-430b-baee-364f8c1b5da1","Type":"ContainerDied","Data":"09a56eec721bef0a0ba0fca6e0c63c2b6f381ded1942568cbb72997492aa4bb0"} Nov 02 23:18:41.636485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:41.636375 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09a56eec721bef0a0ba0fca6e0c63c2b6f381ded1942568cbb72997492aa4bb0" Nov 02 23:18:42.044722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:42.044687 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt"] Nov 02 23:18:42.048223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:42.048195 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb0b6961-k9bxt"] Nov 02 23:18:42.713034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:42.713002 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a42f33-57a4-4ec3-a2ab-6704ea60100e" path="/var/lib/kubelet/pods/a5a42f33-57a4-4ec3-a2ab-6704ea60100e/volumes" Nov 02 23:18:44.805657 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:44.805621 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk"] Nov 02 23:18:44.806120 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:44.805833 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" podUID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" containerName="istio-proxy" containerID="cri-o://2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a" gracePeriod=30 Nov 02 23:18:45.979751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:45.979716 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:18:50.258268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.258239 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:18:50.351873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.351795 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-credential-socket\") pod \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " Nov 02 23:18:50.351873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.351844 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-envoy\") pod \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " Nov 02 23:18:50.352109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.351890 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-data\") pod \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " Nov 02 23:18:50.352109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.351942 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-podinfo\") pod \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " Nov 02 23:18:50.352109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.351960 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-socket\") pod \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " Nov 02 23:18:50.352109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.352050 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-token\" (UniqueName: \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-token\") pod \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " Nov 02 23:18:50.352109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.352058 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-credential-socket" pod "034fd4f3-3ede-4e63-a1c7-61bc593869ac" (UID: "034fd4f3-3ede-4e63-a1c7-61bc593869ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:50.352109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.352097 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istiod-ca-cert\") pod \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " Nov 02 23:18:50.352484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.352191 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-certs\") pod \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " Nov 02 23:18:50.352484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.352260 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-data" pod "034fd4f3-3ede-4e63-a1c7-61bc593869ac" (UID: "034fd4f3-3ede-4e63-a1c7-61bc593869ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:50.352484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.352292 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-kube-api-access-bvvmx\" (UniqueName: \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-kube-api-access-bvvmx\") pod \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\" (UID: \"034fd4f3-3ede-4e63-a1c7-61bc593869ac\") " Nov 02 23:18:50.352484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.352333 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-socket" pod "034fd4f3-3ede-4e63-a1c7-61bc593869ac" (UID: "034fd4f3-3ede-4e63-a1c7-61bc593869ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:50.352484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.352453 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-certs" pod "034fd4f3-3ede-4e63-a1c7-61bc593869ac" (UID: "034fd4f3-3ede-4e63-a1c7-61bc593869ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:50.352737 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.352684 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istiod-ca-cert" pod "034fd4f3-3ede-4e63-a1c7-61bc593869ac" (UID: "034fd4f3-3ede-4e63-a1c7-61bc593869ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:18:50.353104 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.353079 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:50.353266 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.353111 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:50.353266 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.353127 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:50.353266 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.353143 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:50.353266 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.353174 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:50.354320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.354292 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-kube-api-access-bvvmx" pod "034fd4f3-3ede-4e63-a1c7-61bc593869ac" (UID: "034fd4f3-3ede-4e63-a1c7-61bc593869ac"). InnerVolumeSpecName "kube-api-access-bvvmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:50.354424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.354338 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-podinfo" pod "034fd4f3-3ede-4e63-a1c7-61bc593869ac" (UID: "034fd4f3-3ede-4e63-a1c7-61bc593869ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:18:50.354424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.354371 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-token" pod "034fd4f3-3ede-4e63-a1c7-61bc593869ac" (UID: "034fd4f3-3ede-4e63-a1c7-61bc593869ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:18:50.354424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.354387 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-envoy" pod "034fd4f3-3ede-4e63-a1c7-61bc593869ac" (UID: "034fd4f3-3ede-4e63-a1c7-61bc593869ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:18:50.454802 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.454767 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvvmx\" (UniqueName: \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-kube-api-access-bvvmx\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:50.454802 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.454800 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:50.454802 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.454810 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:50.455042 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.454819 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/034fd4f3-3ede-4e63-a1c7-61bc593869ac-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:18:50.723295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.723265 5672 generic.go:356] "Generic (PLEG): container finished" podID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" containerID="2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a" exitCode=0 Nov 02 23:18:50.723519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.723312 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" Nov 02 23:18:50.723519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.723363 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" event={"ID":"034fd4f3-3ede-4e63-a1c7-61bc593869ac","Type":"ContainerDied","Data":"2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a"} Nov 02 23:18:50.723519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.723428 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk" event={"ID":"034fd4f3-3ede-4e63-a1c7-61bc593869ac","Type":"ContainerDied","Data":"c446b460219ac3351cf709879053e1f66a796a9d1cbdb2190354b985c6fa0ee7"} Nov 02 23:18:50.723519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.723457 5672 scope.go:122] "RemoveContainer" containerID="2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a" Nov 02 23:18:50.735541 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.735507 5672 scope.go:122] "RemoveContainer" containerID="2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a" Nov 02 23:18:50.735939 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:18:50.735887 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a\": container with ID starting with 2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a not found: ID does not exist" containerID="2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a" Nov 02 23:18:50.735939 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.735937 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a"} err="failed to get container status \"2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a\": rpc error: code = NotFound desc = could not find container \"2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a\": container with ID starting with 2b0a73ebee616bcbf8f165c3949fd7a2f0e87f9e5c202acf93ad73819138c46a not found: ID does not exist" Nov 02 23:18:50.883292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.883262 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk"] Nov 02 23:18:50.891552 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:50.891528 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-bkjbk"] Nov 02 23:18:52.713596 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:52.713563 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" path="/var/lib/kubelet/pods/034fd4f3-3ede-4e63-a1c7-61bc593869ac/volumes" Nov 02 23:18:56.021604 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:18:56.021554 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:18:56.044377 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:56.044341 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z"] Nov 02 23:18:56.089963 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:56.089935 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e4adb9-njl4z"] Nov 02 23:18:56.717115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:18:56.717062 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f9dfd2-af25-42bc-9bce-0945893921e1" path="/var/lib/kubelet/pods/90f9dfd2-af25-42bc-9bce-0945893921e1/volumes" Nov 02 23:19:26.022353 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:19:26.022245 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:19:27.485697 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:27.485668 5672 scope.go:122] "RemoveContainer" containerID="b67907cb5e15ed4fb55dd502f59be4b73134934ca0e203741e76d8b0951a4f2a" Nov 02 23:19:27.496321 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:27.496289 5672 scope.go:122] "RemoveContainer" containerID="ea7580fd38cd5b8f1ff9f99edafcaa1b319e9366cbb1e9c345603f3eac2dcbdb" Nov 02 23:19:27.505243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:27.505224 5672 scope.go:122] "RemoveContainer" containerID="23ab8e9ed3271b2484fdf2e9d83a62ae47862bba3a6b3f72fcbb25dd32cd92f4" Nov 02 23:19:27.514690 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:27.514666 5672 scope.go:122] "RemoveContainer" containerID="7cc065ff565b4f01f7b0abe3d2dcbadb774e42fb80f13e6355675f07f7d0b25b" Nov 02 23:19:27.522708 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:27.522691 5672 scope.go:122] "RemoveContainer" containerID="ce0ad43b3af7bd32f98b23e1b3d7441e60af934d41155a3e9a3f76dcf3fb8298" Nov 02 23:19:27.555573 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:27.555551 5672 scope.go:122] "RemoveContainer" containerID="ebf53c9032f168bf1b9ede4199c5f95e3963cfe3833dc5d3c1793296639dda7c" Nov 02 23:19:27.564416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:27.564390 5672 scope.go:122] "RemoveContainer" containerID="52448a9625eca0e9bad8c3a9c80b99ed2ceccb4c94afda4f6612c0ade20b2379" Nov 02 23:19:27.576958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:27.576939 5672 scope.go:122] "RemoveContainer" containerID="8fdf98066a2bb3bade155269676767362862161c5189aa49f7b0756d7e551e64" Nov 02 23:19:27.587424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:27.587392 5672 scope.go:122] "RemoveContainer" containerID="095e9e5fd05fff087a453324ba4292269115a4ec84823de98aa2a48ba0e73c44" Nov 02 23:19:27.595868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:27.595845 5672 scope.go:122] "RemoveContainer" containerID="1226f148683b7e04b0d81f9a200909868c4b628c5c77f0d3dcb9998b93045c32" Nov 02 23:19:32.012539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:32.012506 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5"] Nov 02 23:19:32.012998 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:32.012690 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" containerID="cri-o://ae391b5fb0a5f2a6bd3e1fcff6970965cf3c53cb35355db7bae92e1002cc2b05" gracePeriod=30 Nov 02 23:19:32.144770 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:32.144737 5672 generic.go:356] "Generic (PLEG): container finished" podID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerID="a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa" exitCode=0 Nov 02 23:19:32.144945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:32.144800 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" event={"ID":"ccc2dccd-0e58-471e-92c4-9981abf92255","Type":"ContainerDied","Data":"a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa"} Nov 02 23:19:32.145235 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:32.145220 5672 scope.go:122] "RemoveContainer" containerID="a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa" Nov 02 23:19:32.146748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:32.146718 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:32.146875 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:32.146831 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:32.146943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:32.146879 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:33.165201 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:33.163851 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" event={"ID":"ccc2dccd-0e58-471e-92c4-9981abf92255","Type":"ContainerStarted","Data":"5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46"} Nov 02 23:19:33.395309 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:33.395276 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:19:33.395490 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:33.395466 5672 patch_prober.go:29] interesting pod/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:19:33.395549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:33.395518 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 23:19:33.686567 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:33.686532 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd"] Nov 02 23:19:33.686761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:33.686713 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" podUID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" containerName="istio-proxy" containerID="cri-o://d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb" gracePeriod=30 Nov 02 23:19:34.396077 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:34.396041 5672 patch_prober.go:29] interesting pod/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:19:34.396493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:34.396099 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 23:19:35.396317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:35.396281 5672 patch_prober.go:29] interesting pod/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:19:35.396720 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:35.396336 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 23:19:36.396178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:36.396132 5672 patch_prober.go:29] interesting pod/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:19:36.396363 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:36.396205 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 23:19:37.220791 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.220757 5672 generic.go:356] "Generic (PLEG): container finished" podID="62c30e52-e171-45c7-948b-f72af978cd59" containerID="ae391b5fb0a5f2a6bd3e1fcff6970965cf3c53cb35355db7bae92e1002cc2b05" exitCode=0 Nov 02 23:19:37.220982 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.220954 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" event={"ID":"62c30e52-e171-45c7-948b-f72af978cd59","Type":"ContainerDied","Data":"ae391b5fb0a5f2a6bd3e1fcff6970965cf3c53cb35355db7bae92e1002cc2b05"} Nov 02 23:19:37.396421 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.396336 5672 patch_prober.go:29] interesting pod/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:19:37.396421 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.396390 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 23:19:37.469129 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.469102 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:19:37.574142 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574110 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-certs\") pod \"62c30e52-e171-45c7-948b-f72af978cd59\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " Nov 02 23:19:37.574329 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574203 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62c30e52-e171-45c7-948b-f72af978cd59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c30e52-e171-45c7-948b-f72af978cd59-istio-podinfo\") pod \"62c30e52-e171-45c7-948b-f72af978cd59\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " Nov 02 23:19:37.574329 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574266 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-data\") pod \"62c30e52-e171-45c7-948b-f72af978cd59\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " Nov 02 23:19:37.574329 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574309 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-kube-api-access-29zfd\" (UniqueName: \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-kube-api-access-29zfd\") pod \"62c30e52-e171-45c7-948b-f72af978cd59\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " Nov 02 23:19:37.574473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574439 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-certs" pod "62c30e52-e171-45c7-948b-f72af978cd59" (UID: "62c30e52-e171-45c7-948b-f72af978cd59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:37.574473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574462 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-istio-token\" (UniqueName: \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-istio-token\") pod \"62c30e52-e171-45c7-948b-f72af978cd59\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " Nov 02 23:19:37.574563 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574492 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-socket\") pod \"62c30e52-e171-45c7-948b-f72af978cd59\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " Nov 02 23:19:37.574563 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574519 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c30e52-e171-45c7-948b-f72af978cd59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c30e52-e171-45c7-948b-f72af978cd59-istiod-ca-cert\") pod \"62c30e52-e171-45c7-948b-f72af978cd59\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " Nov 02 23:19:37.574675 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574606 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-envoy\") pod \"62c30e52-e171-45c7-948b-f72af978cd59\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " Nov 02 23:19:37.574675 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574639 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-credential-socket\") pod \"62c30e52-e171-45c7-948b-f72af978cd59\" (UID: \"62c30e52-e171-45c7-948b-f72af978cd59\") " Nov 02 23:19:37.574779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574759 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-data" pod "62c30e52-e171-45c7-948b-f72af978cd59" (UID: "62c30e52-e171-45c7-948b-f72af978cd59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:37.574998 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.574961 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c30e52-e171-45c7-948b-f72af978cd59-istiod-ca-cert" pod "62c30e52-e171-45c7-948b-f72af978cd59" (UID: "62c30e52-e171-45c7-948b-f72af978cd59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:37.575248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.575138 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-credential-socket" pod "62c30e52-e171-45c7-948b-f72af978cd59" (UID: "62c30e52-e171-45c7-948b-f72af978cd59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:37.575321 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.575279 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:37.575321 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.575298 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:37.575321 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.575312 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c30e52-e171-45c7-948b-f72af978cd59-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:37.575465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.575354 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:37.575465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.575384 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-socket" pod "62c30e52-e171-45c7-948b-f72af978cd59" (UID: "62c30e52-e171-45c7-948b-f72af978cd59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:37.577484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.577460 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-istio-token" pod "62c30e52-e171-45c7-948b-f72af978cd59" (UID: "62c30e52-e171-45c7-948b-f72af978cd59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:37.577580 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.577546 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-kube-api-access-29zfd" pod "62c30e52-e171-45c7-948b-f72af978cd59" (UID: "62c30e52-e171-45c7-948b-f72af978cd59"). InnerVolumeSpecName "kube-api-access-29zfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:37.577580 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.577547 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-envoy" pod "62c30e52-e171-45c7-948b-f72af978cd59" (UID: "62c30e52-e171-45c7-948b-f72af978cd59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:37.577664 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.577650 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62c30e52-e171-45c7-948b-f72af978cd59-istio-podinfo" pod "62c30e52-e171-45c7-948b-f72af978cd59" (UID: "62c30e52-e171-45c7-948b-f72af978cd59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:37.679299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.679194 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:37.679299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.679236 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:37.679299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.679253 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c30e52-e171-45c7-948b-f72af978cd59-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:37.679299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.679274 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c30e52-e171-45c7-948b-f72af978cd59-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:37.679299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:37.679290 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29zfd\" (UniqueName: \"kubernetes.io/projected/62c30e52-e171-45c7-948b-f72af978cd59-kube-api-access-29zfd\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:38.249510 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:38.249462 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" event={"ID":"62c30e52-e171-45c7-948b-f72af978cd59","Type":"ContainerDied","Data":"146de0c0366d20883bdf0d7b7cdf060add392ec72f7d23dac46d0fee496f0074"} Nov 02 23:19:38.249724 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:38.249538 5672 scope.go:122] "RemoveContainer" containerID="ae391b5fb0a5f2a6bd3e1fcff6970965cf3c53cb35355db7bae92e1002cc2b05" Nov 02 23:19:38.249795 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:38.249730 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5" Nov 02 23:19:38.396474 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:38.396438 5672 patch_prober.go:29] interesting pod/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 23:19:38.396851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:38.396494 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 23:19:38.405617 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:38.405581 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5"] Nov 02 23:19:38.454448 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:38.454411 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-x2mf5"] Nov 02 23:19:38.702715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:38.702684 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:19:38.714975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:38.714950 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c30e52-e171-45c7-948b-f72af978cd59" path="/var/lib/kubelet/pods/62c30e52-e171-45c7-948b-f72af978cd59/volumes" Nov 02 23:19:38.957346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:38.957295 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:19:39.100455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100424 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istiod-ca-cert\") pod \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " Nov 02 23:19:39.100455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100462 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-socket\") pod \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " Nov 02 23:19:39.100668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100492 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-envoy\") pod \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " Nov 02 23:19:39.100668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100546 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-certs\") pod \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " Nov 02 23:19:39.100668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100622 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-data\") pod \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " Nov 02 23:19:39.100848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100683 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-credential-socket\") pod \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " Nov 02 23:19:39.100848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100723 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-kube-api-access-z22f8\" (UniqueName: \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-kube-api-access-z22f8\") pod \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " Nov 02 23:19:39.100848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100728 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-socket" pod "799e0d2a-5dce-470c-a73d-dd9b27258dd2" (UID: "799e0d2a-5dce-470c-a73d-dd9b27258dd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:39.100848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100763 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-token\" (UniqueName: \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-token\") pod \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " Nov 02 23:19:39.100848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100815 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-podinfo\") pod \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\" (UID: \"799e0d2a-5dce-470c-a73d-dd9b27258dd2\") " Nov 02 23:19:39.101098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100931 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-data" pod "799e0d2a-5dce-470c-a73d-dd9b27258dd2" (UID: "799e0d2a-5dce-470c-a73d-dd9b27258dd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:39.101098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100957 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-credential-socket" pod "799e0d2a-5dce-470c-a73d-dd9b27258dd2" (UID: "799e0d2a-5dce-470c-a73d-dd9b27258dd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:39.101098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.100999 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istiod-ca-cert" pod "799e0d2a-5dce-470c-a73d-dd9b27258dd2" (UID: "799e0d2a-5dce-470c-a73d-dd9b27258dd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:39.101636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.101438 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:39.101636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.101464 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:39.101636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.101480 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:39.101636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.101493 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:39.101636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.101601 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-certs" pod "799e0d2a-5dce-470c-a73d-dd9b27258dd2" (UID: "799e0d2a-5dce-470c-a73d-dd9b27258dd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:39.103672 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.103634 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-kube-api-access-z22f8" pod "799e0d2a-5dce-470c-a73d-dd9b27258dd2" (UID: "799e0d2a-5dce-470c-a73d-dd9b27258dd2"). InnerVolumeSpecName "kube-api-access-z22f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:39.103672 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.103642 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-token" pod "799e0d2a-5dce-470c-a73d-dd9b27258dd2" (UID: "799e0d2a-5dce-470c-a73d-dd9b27258dd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:39.103993 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.103973 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-podinfo" pod "799e0d2a-5dce-470c-a73d-dd9b27258dd2" (UID: "799e0d2a-5dce-470c-a73d-dd9b27258dd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:19:39.104326 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.104305 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-envoy" pod "799e0d2a-5dce-470c-a73d-dd9b27258dd2" (UID: "799e0d2a-5dce-470c-a73d-dd9b27258dd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:19:39.203215 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.203190 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z22f8\" (UniqueName: \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-kube-api-access-z22f8\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:39.203215 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.203213 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:39.203391 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.203223 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:39.203391 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.203232 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:39.203391 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.203242 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799e0d2a-5dce-470c-a73d-dd9b27258dd2-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:39.257695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.257664 5672 generic.go:356] "Generic (PLEG): container finished" podID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" containerID="d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb" exitCode=0 Nov 02 23:19:39.257884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.257720 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" Nov 02 23:19:39.257884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.257747 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" event={"ID":"799e0d2a-5dce-470c-a73d-dd9b27258dd2","Type":"ContainerDied","Data":"d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb"} Nov 02 23:19:39.257884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.257806 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd" event={"ID":"799e0d2a-5dce-470c-a73d-dd9b27258dd2","Type":"ContainerDied","Data":"8d9f7098a2aee5efb7cd638b744c12b8728b9c5aa6543cdb91cdb2e20ad52c8d"} Nov 02 23:19:39.257884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.257834 5672 scope.go:122] "RemoveContainer" containerID="d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb" Nov 02 23:19:39.269507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.269485 5672 scope.go:122] "RemoveContainer" containerID="d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb" Nov 02 23:19:39.269779 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:19:39.269757 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb\": container with ID starting with d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb not found: ID does not exist" containerID="d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb" Nov 02 23:19:39.269827 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.269791 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb"} err="failed to get container status \"d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb\": rpc error: code = NotFound desc = could not find container \"d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb\": container with ID starting with d910f487ba929dc273836b629c42d8c3caea76605782ae1baef06c15464561eb not found: ID does not exist" Nov 02 23:19:39.401030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.401002 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:19:39.401418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.401372 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:19:39.402052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.402030 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:19:39.412748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.412719 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd"] Nov 02 23:19:39.421464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:39.421444 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-mckvd"] Nov 02 23:19:40.713399 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:40.713371 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" path="/var/lib/kubelet/pods/799e0d2a-5dce-470c-a73d-dd9b27258dd2/volumes" Nov 02 23:19:43.717021 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.716984 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627"] Nov 02 23:19:43.717561 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717546 5672 cpu_manager.go:414] "containerMap: removing container" podUID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" containerName="istio-proxy" Nov 02 23:19:43.717606 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717562 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" containerName="istio-proxy" Nov 02 23:19:43.717606 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717581 5672 cpu_manager.go:414] "containerMap: removing container" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" Nov 02 23:19:43.717606 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717587 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" Nov 02 23:19:43.717701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717615 5672 cpu_manager.go:414] "containerMap: removing container" podUID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" containerName="istio-proxy" Nov 02 23:19:43.717701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717620 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" containerName="istio-proxy" Nov 02 23:19:43.717701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717630 5672 cpu_manager.go:414] "containerMap: removing container" podUID="cdfd4a95-9541-430b-baee-364f8c1b5da1" containerName="revoke-keys" Nov 02 23:19:43.717701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717635 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdfd4a95-9541-430b-baee-364f8c1b5da1" containerName="revoke-keys" Nov 02 23:19:43.717815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717727 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="034fd4f3-3ede-4e63-a1c7-61bc593869ac" containerName="istio-proxy" Nov 02 23:19:43.717815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717735 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdfd4a95-9541-430b-baee-364f8c1b5da1" containerName="revoke-keys" Nov 02 23:19:43.717815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717745 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="799e0d2a-5dce-470c-a73d-dd9b27258dd2" containerName="istio-proxy" Nov 02 23:19:43.717815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.717752 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c30e52-e171-45c7-948b-f72af978cd59" containerName="istio-proxy" Nov 02 23:19:43.721516 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.721496 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.742457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.742426 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.742601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.742576 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.742659 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.742604 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.742732 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.742718 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.742834 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.742818 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ef579f-a819-4988-ab22-eafd9da5e6b7-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.742872 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.742847 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.742949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.742935 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6nzl\" (UniqueName: \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-kube-api-access-h6nzl\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.742989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.742967 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.743036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.742989 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.770668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.770645 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e953257b-openshift-default-dockercfg-9qzkk\"" Nov 02 23:19:43.821288 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.821251 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627"] Nov 02 23:19:43.844202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.844135 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.844449 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.844427 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.845036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.845017 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.845652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.845631 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ef579f-a819-4988-ab22-eafd9da5e6b7-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.846805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.846785 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.846982 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.846962 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6nzl\" (UniqueName: \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-kube-api-access-h6nzl\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.847066 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.847008 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.847066 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.847047 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.847221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.847098 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.847221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.846455 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ef579f-a819-4988-ab22-eafd9da5e6b7-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.847329 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.847100 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.847632 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.847605 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.847704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.847610 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.847704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.845505 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.848416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.848374 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.848872 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.848851 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.900366 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.900334 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:43.900502 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:43.900372 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6nzl\" (UniqueName: \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-kube-api-access-h6nzl\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-25627\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:44.034513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:44.034477 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:44.451832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:44.451807 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627"] Nov 02 23:19:44.453575 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:19:44.453533 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ef579f_a819_4988_ab22_eafd9da5e6b7.slice/crio-8adb283fb28315e4ec018cdf9835ff847b9aac3b4ddadb48d2c85e9d3d2f1317 WatchSource:0}: Error finding container 8adb283fb28315e4ec018cdf9835ff847b9aac3b4ddadb48d2c85e9d3d2f1317: Status 404 returned error can't find the container with id 8adb283fb28315e4ec018cdf9835ff847b9aac3b4ddadb48d2c85e9d3d2f1317 Nov 02 23:19:44.456087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:44.456051 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:44.456218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:44.456130 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:44.456218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:44.456190 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:45.309432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:45.309394 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" event={"ID":"28ef579f-a819-4988-ab22-eafd9da5e6b7","Type":"ContainerStarted","Data":"f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60"} Nov 02 23:19:45.309918 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:45.309458 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" event={"ID":"28ef579f-a819-4988-ab22-eafd9da5e6b7","Type":"ContainerStarted","Data":"8adb283fb28315e4ec018cdf9835ff847b9aac3b4ddadb48d2c85e9d3d2f1317"} Nov 02 23:19:45.466795 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:45.466742 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" podStartSLOduration=2.466728556 podStartE2EDuration="2.466728556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:45.463856278 +0000 UTC m=+1578.871168120" watchObservedRunningTime="2026-11-02 23:19:45.466728556 +0000 UTC m=+1578.874040386" Nov 02 23:19:46.050173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:46.040831 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:46.050173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:46.040960 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-25627 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 23:19:46.050173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:46.041002 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" podUID="28ef579f-a819-4988-ab22-eafd9da5e6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 23:19:47.037564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:47.037528 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-25627 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 23:19:47.038135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:47.038100 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" podUID="28ef579f-a819-4988-ab22-eafd9da5e6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 23:19:48.046170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:48.045258 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-25627 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 23:19:48.046170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:48.045335 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" podUID="28ef579f-a819-4988-ab22-eafd9da5e6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 23:19:49.045173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:49.037526 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-25627 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 23:19:49.045173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:49.037606 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" podUID="28ef579f-a819-4988-ab22-eafd9da5e6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 23:19:50.050232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:50.036303 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-25627 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 23:19:50.050232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:50.036380 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" podUID="28ef579f-a819-4988-ab22-eafd9da5e6b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 23:19:51.039058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:51.039027 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:51.039539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:51.039519 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:51.040134 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:51.040116 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:19:52.638980 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:52.638940 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc"] Nov 02 23:19:54.046228 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:54.046197 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l"] Nov 02 23:19:54.050673 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:54.050644 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80e60c3c-lnp5l"] Nov 02 23:19:54.713972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:54.713937 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e738fd68-52f3-4f57-b0a1-a4ab12c091ec" path="/var/lib/kubelet/pods/e738fd68-52f3-4f57-b0a1-a4ab12c091ec/volumes" Nov 02 23:19:56.020830 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:19:56.020787 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:19:57.640815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:57.640744 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" podUID="0c4fd533-9517-408a-b122-1f3287772b30" containerName="payload-processing" containerID="cri-o://5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada" gracePeriod=25 Nov 02 23:19:57.934505 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:57.934472 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:19:58.049035 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.048995 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4fd533-9517-408a-b122-1f3287772b30-kube-api-access-p8wmd\" (UniqueName: \"kubernetes.io/projected/0c4fd533-9517-408a-b122-1f3287772b30-kube-api-access-p8wmd\") pod \"0c4fd533-9517-408a-b122-1f3287772b30\" (UID: \"0c4fd533-9517-408a-b122-1f3287772b30\") " Nov 02 23:19:58.049213 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.049183 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4fd533-9517-408a-b122-1f3287772b30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4fd533-9517-408a-b122-1f3287772b30-plugins-config-volume\") pod \"0c4fd533-9517-408a-b122-1f3287772b30\" (UID: \"0c4fd533-9517-408a-b122-1f3287772b30\") " Nov 02 23:19:58.049602 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.049574 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4fd533-9517-408a-b122-1f3287772b30-plugins-config-volume" pod "0c4fd533-9517-408a-b122-1f3287772b30" (UID: "0c4fd533-9517-408a-b122-1f3287772b30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:19:58.049708 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.049697 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4fd533-9517-408a-b122-1f3287772b30-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:58.051340 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.051315 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4fd533-9517-408a-b122-1f3287772b30-kube-api-access-p8wmd" pod "0c4fd533-9517-408a-b122-1f3287772b30" (UID: "0c4fd533-9517-408a-b122-1f3287772b30"). InnerVolumeSpecName "kube-api-access-p8wmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:19:58.151236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.151137 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8wmd\" (UniqueName: \"kubernetes.io/projected/0c4fd533-9517-408a-b122-1f3287772b30-kube-api-access-p8wmd\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:19:58.428276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.428144 5672 generic.go:356] "Generic (PLEG): container finished" podID="0c4fd533-9517-408a-b122-1f3287772b30" containerID="5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada" exitCode=0 Nov 02 23:19:58.428276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.428190 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" event={"ID":"0c4fd533-9517-408a-b122-1f3287772b30","Type":"ContainerDied","Data":"5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada"} Nov 02 23:19:58.428276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.428229 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" Nov 02 23:19:58.428546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.428336 5672 scope.go:122] "RemoveContainer" containerID="5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada" Nov 02 23:19:58.428546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.428332 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc" event={"ID":"0c4fd533-9517-408a-b122-1f3287772b30","Type":"ContainerDied","Data":"f8697bcfc51df9a60868e18b99c389a6012c02ee3811132d373d37ae6ccc6850"} Nov 02 23:19:58.440851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.440829 5672 scope.go:122] "RemoveContainer" containerID="5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada" Nov 02 23:19:58.441248 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:19:58.441222 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada\": container with ID starting with 5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada not found: ID does not exist" containerID="5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada" Nov 02 23:19:58.441372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.441253 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada"} err="failed to get container status \"5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada\": rpc error: code = NotFound desc = could not find container \"5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada\": container with ID starting with 5d9ab248e61dbda2a3d21b47908cbd407c2c6ddea681c31a1804246984581ada not found: ID does not exist" Nov 02 23:19:58.582980 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.582930 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc"] Nov 02 23:19:58.628657 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.628622 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ede388-54df75d6d9-rmmvc"] Nov 02 23:19:58.713048 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.712964 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4fd533-9517-408a-b122-1f3287772b30" path="/var/lib/kubelet/pods/0c4fd533-9517-408a-b122-1f3287772b30/volumes" Nov 02 23:19:58.843750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.843715 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg"] Nov 02 23:19:58.844304 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.844288 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0c4fd533-9517-408a-b122-1f3287772b30" containerName="payload-processing" Nov 02 23:19:58.844367 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.844304 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4fd533-9517-408a-b122-1f3287772b30" containerName="payload-processing" Nov 02 23:19:58.844418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.844409 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4fd533-9517-408a-b122-1f3287772b30" containerName="payload-processing" Nov 02 23:19:58.847839 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.847817 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:58.930671 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.930637 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg"] Nov 02 23:19:58.960828 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.960794 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:58.961010 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.960856 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:58.961090 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.961068 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a206e651-2148-4971-9c30-8ed88959a6b4-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:58.961176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.961128 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:58.961321 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.961301 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9wg\" (UniqueName: \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-kube-api-access-qp9wg\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:58.961375 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.961342 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a206e651-2148-4971-9c30-8ed88959a6b4-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:58.961452 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.961431 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:58.961545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.961501 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:58.961588 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:58.961538 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.063135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.063092 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9wg\" (UniqueName: \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-kube-api-access-qp9wg\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.063362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.063177 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a206e651-2148-4971-9c30-8ed88959a6b4-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.063362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.063239 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.063362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.063285 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.063362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.063321 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.063362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.063356 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.063692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.063399 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.063692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.063520 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a206e651-2148-4971-9c30-8ed88959a6b4-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.063692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.063560 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.063982 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.063958 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.064126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.064028 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a206e651-2148-4971-9c30-8ed88959a6b4-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.064242 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.064131 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.064301 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.064242 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.064462 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.064438 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.066190 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.066123 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.066481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.066456 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a206e651-2148-4971-9c30-8ed88959a6b4-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.116639 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.116605 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.116843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.116824 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9wg\" (UniqueName: \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-kube-api-access-qp9wg\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.161539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.161505 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:19:59.586124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.586090 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg"] Nov 02 23:19:59.587739 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:19:59.587707 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda206e651_2148_4971_9c30_8ed88959a6b4.slice/crio-e745381cfb598ad2131a51be9ef502d484d7a04e40909e35d2a8f486c8af11a2 WatchSource:0}: Error finding container e745381cfb598ad2131a51be9ef502d484d7a04e40909e35d2a8f486c8af11a2: Status 404 returned error can't find the container with id e745381cfb598ad2131a51be9ef502d484d7a04e40909e35d2a8f486c8af11a2 Nov 02 23:19:59.591036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.591002 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:59.591174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.591087 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:19:59.591281 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:19:59.591179 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:20:00.463409 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:00.463358 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" event={"ID":"a206e651-2148-4971-9c30-8ed88959a6b4","Type":"ContainerStarted","Data":"b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399"} Nov 02 23:20:00.463929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:00.463432 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" event={"ID":"a206e651-2148-4971-9c30-8ed88959a6b4","Type":"ContainerStarted","Data":"e745381cfb598ad2131a51be9ef502d484d7a04e40909e35d2a8f486c8af11a2"} Nov 02 23:20:00.621607 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:00.621546 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" podStartSLOduration=2.6215324190000002 podStartE2EDuration="2.621532419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:00.621510068 +0000 UTC m=+1594.028821899" watchObservedRunningTime="2026-11-02 23:20:00.621532419 +0000 UTC m=+1594.028844249" Nov 02 23:20:01.162693 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:01.162589 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:20:01.164698 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:01.164673 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 23:20:01.164820 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:01.164723 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" podUID="a206e651-2148-4971-9c30-8ed88959a6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 23:20:02.162074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:02.162037 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 23:20:02.162620 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:02.162095 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" podUID="a206e651-2148-4971-9c30-8ed88959a6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 23:20:03.162700 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:03.162659 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 23:20:03.163129 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:03.162738 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" podUID="a206e651-2148-4971-9c30-8ed88959a6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 23:20:04.169175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:04.162761 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 23:20:04.169175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:04.162834 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" podUID="a206e651-2148-4971-9c30-8ed88959a6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 23:20:05.161973 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:05.161938 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 23:20:05.162165 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:05.162007 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" podUID="a206e651-2148-4971-9c30-8ed88959a6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 23:20:06.167764 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:06.167731 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:20:06.168251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:06.168051 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:20:06.168878 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:06.168857 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:20:12.437828 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:12.437792 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc"] Nov 02 23:20:13.958868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:13.958832 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn"] Nov 02 23:20:13.962021 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:13.962001 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.057939 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.057909 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn"] Nov 02 23:20:14.085445 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.085409 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f92e90-6561-40df-89d2-754922005318-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.085584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.085498 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.085584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.085556 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.085702 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.085589 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.085760 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.085740 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.085851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.085836 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f92e90-6561-40df-89d2-754922005318-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.085945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.085931 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr4pg\" (UniqueName: \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-kube-api-access-jr4pg\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.086014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.085969 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.086014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.086004 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.187568 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.187532 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f92e90-6561-40df-89d2-754922005318-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.187568 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.187570 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.187824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.187608 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.187824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.187637 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.187824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.187673 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.187824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.187727 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f92e90-6561-40df-89d2-754922005318-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.187824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.187791 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr4pg\" (UniqueName: \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-kube-api-access-jr4pg\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.188095 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.187825 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.188095 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.187856 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.188481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.188414 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.192290 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.188116 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.192290 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.188603 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.192290 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.188623 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.192290 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.188728 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f92e90-6561-40df-89d2-754922005318-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.201537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.201504 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.202905 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.202880 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f92e90-6561-40df-89d2-754922005318-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.241475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.241444 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.241609 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.241527 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr4pg\" (UniqueName: \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-kube-api-access-jr4pg\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.275863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.275834 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:14.693199 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.693169 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn"] Nov 02 23:20:14.694544 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:20:14.694514 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f92e90_6561_40df_89d2_754922005318.slice/crio-ddeffdddcc483ba98c893324efd7915286a62fbde9cef1dc02d4a7682f5cee41 WatchSource:0}: Error finding container ddeffdddcc483ba98c893324efd7915286a62fbde9cef1dc02d4a7682f5cee41: Status 404 returned error can't find the container with id ddeffdddcc483ba98c893324efd7915286a62fbde9cef1dc02d4a7682f5cee41 Nov 02 23:20:14.697425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.697393 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:20:14.697507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.697455 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:20:14.697507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:14.697484 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:20:15.577411 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:15.577367 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" event={"ID":"21f92e90-6561-40df-89d2-754922005318","Type":"ContainerStarted","Data":"2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7"} Nov 02 23:20:15.577837 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:15.577439 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" event={"ID":"21f92e90-6561-40df-89d2-754922005318","Type":"ContainerStarted","Data":"ddeffdddcc483ba98c893324efd7915286a62fbde9cef1dc02d4a7682f5cee41"} Nov 02 23:20:15.799661 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:15.799606 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" podStartSLOduration=2.7995901930000002 podStartE2EDuration="2.799590193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:15.751723867 +0000 UTC m=+1609.159035698" watchObservedRunningTime="2026-11-02 23:20:15.799590193 +0000 UTC m=+1609.206902023" Nov 02 23:20:16.276649 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:16.276616 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:16.278213 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:16.278187 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 23:20:16.278341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:16.278237 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" podUID="21f92e90-6561-40df-89d2-754922005318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 23:20:17.276264 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:17.276222 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 23:20:17.276733 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:17.276330 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" podUID="21f92e90-6561-40df-89d2-754922005318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 23:20:18.277171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:18.276723 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 23:20:18.277171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:18.276811 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" podUID="21f92e90-6561-40df-89d2-754922005318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 23:20:19.276469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:19.276435 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 23:20:19.276694 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:19.276490 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" podUID="21f92e90-6561-40df-89d2-754922005318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 23:20:20.276717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:20.276676 5672 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 23:20:20.277113 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:20.276744 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" podUID="21f92e90-6561-40df-89d2-754922005318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 23:20:21.280665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:21.280634 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:21.281111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:21.281098 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:21.281793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:21.281771 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:20:24.262513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:24.262481 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:20:26.022101 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:20:26.022065 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:20:27.459007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.458964 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" podUID="2404095a-13ed-45fc-8470-79dc7da99cd7" containerName="main" containerID="cri-o://5630a83f58a277089e7e0b582f8de788bd251acdb1bc08e2f0cf2a84d8ae9b35" gracePeriod=45 Nov 02 23:20:27.691351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.691322 5672 scope.go:122] "RemoveContainer" containerID="241cfde8d10f5ac95dce50dfd92c077acbe650c4a03c013e6a454a9cadb853ee" Nov 02 23:20:27.691903 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.691862 5672 generic.go:356] "Generic (PLEG): container finished" podID="2404095a-13ed-45fc-8470-79dc7da99cd7" containerID="5630a83f58a277089e7e0b582f8de788bd251acdb1bc08e2f0cf2a84d8ae9b35" exitCode=0 Nov 02 23:20:27.692131 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.692108 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" event={"ID":"2404095a-13ed-45fc-8470-79dc7da99cd7","Type":"ContainerDied","Data":"5630a83f58a277089e7e0b582f8de788bd251acdb1bc08e2f0cf2a84d8ae9b35"} Nov 02 23:20:27.703604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.703584 5672 scope.go:122] "RemoveContainer" containerID="4b93176eb042560bb4b3df90c7ce021fe358cff00a0a959031f73aecdfc7701b" Nov 02 23:20:27.739487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.739469 5672 scope.go:122] "RemoveContainer" containerID="e3f0553f6403b9e6c7e3a16b50796594f52216f03bd7f27f51c18fa210be9ff8" Nov 02 23:20:27.740688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.740669 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:20:27.830948 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.830909 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-home\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-home\") pod \"2404095a-13ed-45fc-8470-79dc7da99cd7\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " Nov 02 23:20:27.831127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.830995 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2404095a-13ed-45fc-8470-79dc7da99cd7-kube-api-access-57g2t\" (UniqueName: \"kubernetes.io/projected/2404095a-13ed-45fc-8470-79dc7da99cd7-kube-api-access-57g2t\") pod \"2404095a-13ed-45fc-8470-79dc7da99cd7\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " Nov 02 23:20:27.831127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.831043 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2404095a-13ed-45fc-8470-79dc7da99cd7-tls-certs\" (UniqueName: \"kubernetes.io/secret/2404095a-13ed-45fc-8470-79dc7da99cd7-tls-certs\") pod \"2404095a-13ed-45fc-8470-79dc7da99cd7\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " Nov 02 23:20:27.831127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.831063 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-dshm\") pod \"2404095a-13ed-45fc-8470-79dc7da99cd7\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " Nov 02 23:20:27.831278 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.831224 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-tmp-dir\") pod \"2404095a-13ed-45fc-8470-79dc7da99cd7\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " Nov 02 23:20:27.831278 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.831239 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-home" pod "2404095a-13ed-45fc-8470-79dc7da99cd7" (UID: "2404095a-13ed-45fc-8470-79dc7da99cd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:27.831278 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.831274 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-model-cache\") pod \"2404095a-13ed-45fc-8470-79dc7da99cd7\" (UID: \"2404095a-13ed-45fc-8470-79dc7da99cd7\") " Nov 02 23:20:27.831525 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.831465 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-tmp-dir" pod "2404095a-13ed-45fc-8470-79dc7da99cd7" (UID: "2404095a-13ed-45fc-8470-79dc7da99cd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:27.831666 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.831521 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-model-cache" pod "2404095a-13ed-45fc-8470-79dc7da99cd7" (UID: "2404095a-13ed-45fc-8470-79dc7da99cd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:27.831789 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.831774 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:27.831830 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.831793 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:27.831830 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.831803 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:27.833228 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.833210 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2404095a-13ed-45fc-8470-79dc7da99cd7-kube-api-access-57g2t" pod "2404095a-13ed-45fc-8470-79dc7da99cd7" (UID: "2404095a-13ed-45fc-8470-79dc7da99cd7"). InnerVolumeSpecName "kube-api-access-57g2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:27.833590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.833566 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2404095a-13ed-45fc-8470-79dc7da99cd7-tls-certs" pod "2404095a-13ed-45fc-8470-79dc7da99cd7" (UID: "2404095a-13ed-45fc-8470-79dc7da99cd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:20:27.833590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.833578 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-dshm" pod "2404095a-13ed-45fc-8470-79dc7da99cd7" (UID: "2404095a-13ed-45fc-8470-79dc7da99cd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:20:27.933427 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.933392 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57g2t\" (UniqueName: \"kubernetes.io/projected/2404095a-13ed-45fc-8470-79dc7da99cd7-kube-api-access-57g2t\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:27.933427 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.933424 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2404095a-13ed-45fc-8470-79dc7da99cd7-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:27.933427 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:27.933433 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2404095a-13ed-45fc-8470-79dc7da99cd7-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:28.701113 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:28.701082 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" Nov 02 23:20:28.701113 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:28.701101 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc" event={"ID":"2404095a-13ed-45fc-8470-79dc7da99cd7","Type":"ContainerDied","Data":"76080c20829fcc0829f7689f974c5fb6ff4f11a793d3770e70369a692410cd66"} Nov 02 23:20:28.701650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:28.701168 5672 scope.go:122] "RemoveContainer" containerID="5630a83f58a277089e7e0b582f8de788bd251acdb1bc08e2f0cf2a84d8ae9b35" Nov 02 23:20:28.854725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:28.854695 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc"] Nov 02 23:20:28.901200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:28.901166 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e953257b/e2e-lifecycle-model-e953257b-kserve-5c6944ccd5-p9lnc"] Nov 02 23:20:30.713542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:30.713508 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2404095a-13ed-45fc-8470-79dc7da99cd7" path="/var/lib/kubelet/pods/2404095a-13ed-45fc-8470-79dc7da99cd7/volumes" Nov 02 23:20:39.556741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:39.556649 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs"] Nov 02 23:20:39.656168 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:39.656120 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44"] Nov 02 23:20:44.557432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:44.557387 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" podUID="39b34d72-b9df-4e1e-a0c1-a5d620060bfb" containerName="payload-processing" containerID="cri-o://5e2c5684025a5bb1a58104176133aa22c64400680afcfa3076b261e3f19ec36a" gracePeriod=25 Nov 02 23:20:44.656963 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:44.656911 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" podUID="bd54c36b-d57e-46dc-953d-e6b446d17007" containerName="payload-pre-processing" containerID="cri-o://376c4a12d3d362614152cf58b16a25987d72b69068a19d57f87e6901821967a7" gracePeriod=25 Nov 02 23:20:44.824283 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:44.824194 5672 generic.go:356] "Generic (PLEG): container finished" podID="39b34d72-b9df-4e1e-a0c1-a5d620060bfb" containerID="5e2c5684025a5bb1a58104176133aa22c64400680afcfa3076b261e3f19ec36a" exitCode=0 Nov 02 23:20:44.824456 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:44.824344 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" event={"ID":"39b34d72-b9df-4e1e-a0c1-a5d620060bfb","Type":"ContainerDied","Data":"5e2c5684025a5bb1a58104176133aa22c64400680afcfa3076b261e3f19ec36a"} Nov 02 23:20:44.827089 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:44.827064 5672 generic.go:356] "Generic (PLEG): container finished" podID="bd54c36b-d57e-46dc-953d-e6b446d17007" containerID="376c4a12d3d362614152cf58b16a25987d72b69068a19d57f87e6901821967a7" exitCode=0 Nov 02 23:20:44.827204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:44.827121 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" event={"ID":"bd54c36b-d57e-46dc-953d-e6b446d17007","Type":"ContainerDied","Data":"376c4a12d3d362614152cf58b16a25987d72b69068a19d57f87e6901821967a7"} Nov 02 23:20:44.941486 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:44.941451 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:20:44.985237 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:44.985208 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:20:45.005683 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.005650 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-kube-api-access-p8n8d\" (UniqueName: \"kubernetes.io/projected/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-kube-api-access-p8n8d\") pod \"39b34d72-b9df-4e1e-a0c1-a5d620060bfb\" (UID: \"39b34d72-b9df-4e1e-a0c1-a5d620060bfb\") " Nov 02 23:20:45.005875 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.005737 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-plugins-config-volume\") pod \"39b34d72-b9df-4e1e-a0c1-a5d620060bfb\" (UID: \"39b34d72-b9df-4e1e-a0c1-a5d620060bfb\") " Nov 02 23:20:45.006187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.006161 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-plugins-config-volume" pod "39b34d72-b9df-4e1e-a0c1-a5d620060bfb" (UID: "39b34d72-b9df-4e1e-a0c1-a5d620060bfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:45.006356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.006337 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:45.007934 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.007902 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-kube-api-access-p8n8d" pod "39b34d72-b9df-4e1e-a0c1-a5d620060bfb" (UID: "39b34d72-b9df-4e1e-a0c1-a5d620060bfb"). InnerVolumeSpecName "kube-api-access-p8n8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:45.107185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.107084 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd54c36b-d57e-46dc-953d-e6b446d17007-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd54c36b-d57e-46dc-953d-e6b446d17007-plugins-config-volume\") pod \"bd54c36b-d57e-46dc-953d-e6b446d17007\" (UID: \"bd54c36b-d57e-46dc-953d-e6b446d17007\") " Nov 02 23:20:45.107354 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.107241 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd54c36b-d57e-46dc-953d-e6b446d17007-kube-api-access-m2b2n\" (UniqueName: \"kubernetes.io/projected/bd54c36b-d57e-46dc-953d-e6b446d17007-kube-api-access-m2b2n\") pod \"bd54c36b-d57e-46dc-953d-e6b446d17007\" (UID: \"bd54c36b-d57e-46dc-953d-e6b446d17007\") " Nov 02 23:20:45.107463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.107440 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd54c36b-d57e-46dc-953d-e6b446d17007-plugins-config-volume" pod "bd54c36b-d57e-46dc-953d-e6b446d17007" (UID: "bd54c36b-d57e-46dc-953d-e6b446d17007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:20:45.107715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.107701 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd54c36b-d57e-46dc-953d-e6b446d17007-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:45.107762 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.107718 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8n8d\" (UniqueName: \"kubernetes.io/projected/39b34d72-b9df-4e1e-a0c1-a5d620060bfb-kube-api-access-p8n8d\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:45.109343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.109321 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd54c36b-d57e-46dc-953d-e6b446d17007-kube-api-access-m2b2n" pod "bd54c36b-d57e-46dc-953d-e6b446d17007" (UID: "bd54c36b-d57e-46dc-953d-e6b446d17007"). InnerVolumeSpecName "kube-api-access-m2b2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:20:45.209024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.208983 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2b2n\" (UniqueName: \"kubernetes.io/projected/bd54c36b-d57e-46dc-953d-e6b446d17007-kube-api-access-m2b2n\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:20:45.836189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.836084 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" Nov 02 23:20:45.836189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.836127 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44" event={"ID":"bd54c36b-d57e-46dc-953d-e6b446d17007","Type":"ContainerDied","Data":"c7451632679e3953b73841523261a14e1612e35506ea54d1a144f1629d625702"} Nov 02 23:20:45.836721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.836228 5672 scope.go:122] "RemoveContainer" containerID="376c4a12d3d362614152cf58b16a25987d72b69068a19d57f87e6901821967a7" Nov 02 23:20:45.837834 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.837809 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" Nov 02 23:20:45.837834 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.837822 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs" event={"ID":"39b34d72-b9df-4e1e-a0c1-a5d620060bfb","Type":"ContainerDied","Data":"804a8fcc0364815ce2b91ab6b2daed96c69b31e07c42f552908e59ec8cf16cc8"} Nov 02 23:20:45.848356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.848337 5672 scope.go:122] "RemoveContainer" containerID="5e2c5684025a5bb1a58104176133aa22c64400680afcfa3076b261e3f19ec36a" Nov 02 23:20:45.989452 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.989410 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs"] Nov 02 23:20:45.993212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:45.993180 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79385a-c75d5f567-hphqs"] Nov 02 23:20:46.095866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:46.095770 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44"] Nov 02 23:20:46.098949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:46.098917 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79385a-7c66d77c7f-r2h44"] Nov 02 23:20:46.721059 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:46.721018 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b34d72-b9df-4e1e-a0c1-a5d620060bfb" path="/var/lib/kubelet/pods/39b34d72-b9df-4e1e-a0c1-a5d620060bfb/volumes" Nov 02 23:20:46.721524 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:20:46.721508 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd54c36b-d57e-46dc-953d-e6b446d17007" path="/var/lib/kubelet/pods/bd54c36b-d57e-46dc-953d-e6b446d17007/volumes" Nov 02 23:20:56.021228 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:20:56.021187 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:21:05.044022 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:05.043988 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276"] Nov 02 23:21:05.090651 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:05.090623 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c38777-vd276"] Nov 02 23:21:06.712828 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:06.712787 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb3d538-3424-4b33-98d2-47f9d6903d98" path="/var/lib/kubelet/pods/4eb3d538-3424-4b33-98d2-47f9d6903d98/volumes" Nov 02 23:21:09.750069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:09.750033 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:21:26.021430 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:21:26.021390 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:21:27.786798 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:27.786763 5672 scope.go:122] "RemoveContainer" containerID="1b7286163369779e72cc6b285191dbeba6da78906531a3c4ed9cbdda4b8f352a" Nov 02 23:21:36.869558 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:36.869520 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx"] Nov 02 23:21:38.444629 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.444593 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws"] Nov 02 23:21:38.445356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.445331 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bd54c36b-d57e-46dc-953d-e6b446d17007" containerName="payload-pre-processing" Nov 02 23:21:38.445356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.445355 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd54c36b-d57e-46dc-953d-e6b446d17007" containerName="payload-pre-processing" Nov 02 23:21:38.445537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.445404 5672 cpu_manager.go:414] "containerMap: removing container" podUID="39b34d72-b9df-4e1e-a0c1-a5d620060bfb" containerName="payload-processing" Nov 02 23:21:38.445537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.445413 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b34d72-b9df-4e1e-a0c1-a5d620060bfb" containerName="payload-processing" Nov 02 23:21:38.445537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.445428 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2404095a-13ed-45fc-8470-79dc7da99cd7" containerName="main" Nov 02 23:21:38.445537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.445433 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2404095a-13ed-45fc-8470-79dc7da99cd7" containerName="main" Nov 02 23:21:38.445537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.445537 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b34d72-b9df-4e1e-a0c1-a5d620060bfb" containerName="payload-processing" Nov 02 23:21:38.445692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.445552 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2404095a-13ed-45fc-8470-79dc7da99cd7" containerName="main" Nov 02 23:21:38.445692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.445562 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd54c36b-d57e-46dc-953d-e6b446d17007" containerName="payload-pre-processing" Nov 02 23:21:38.450421 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.450400 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:38.519214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.519006 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws"] Nov 02 23:21:38.563171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.555785 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:21:38.563171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.555892 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:21:38.563171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.555969 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:21:38.563171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.555997 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:21:38.612490 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.612452 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-nmgws\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:38.612696 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.612599 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2gm\" (UniqueName: \"kubernetes.io/projected/89195d4e-ffa0-4231-95d8-5982b1aa34f0-kube-api-access-2k2gm\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-nmgws\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:38.612794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.612768 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-nmgws\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:38.713453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.713380 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2gm\" (UniqueName: \"kubernetes.io/projected/89195d4e-ffa0-4231-95d8-5982b1aa34f0-kube-api-access-2k2gm\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-nmgws\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:38.713453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.713446 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-nmgws\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:38.713622 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.713529 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-nmgws\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:38.714122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.714099 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-nmgws\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:38.715880 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.715852 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-nmgws\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:38.767557 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:38.767521 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2gm\" (UniqueName: \"kubernetes.io/projected/89195d4e-ffa0-4231-95d8-5982b1aa34f0-kube-api-access-2k2gm\") pod \"maas-api-revoke-keys-e2e-mt-e953257b-nmgws\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:39.063978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:39.063948 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:39.444432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:39.444402 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws"] Nov 02 23:21:39.446537 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:21:39.446497 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89195d4e_ffa0_4231_95d8_5982b1aa34f0.slice/crio-5586e8b86d0779e82a5c97dc6fd792c307c5fd7f2f4fc0796cba3eba28a9e57b WatchSource:0}: Error finding container 5586e8b86d0779e82a5c97dc6fd792c307c5fd7f2f4fc0796cba3eba28a9e57b: Status 404 returned error can't find the container with id 5586e8b86d0779e82a5c97dc6fd792c307c5fd7f2f4fc0796cba3eba28a9e57b Nov 02 23:21:40.319870 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:40.319835 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" event={"ID":"89195d4e-ffa0-4231-95d8-5982b1aa34f0","Type":"ContainerStarted","Data":"47dff56f8bfafc357320da889739c681c35de59727a9b6ffb58ff8893ac9cd58"} Nov 02 23:21:40.320038 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:40.319894 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" event={"ID":"89195d4e-ffa0-4231-95d8-5982b1aa34f0","Type":"ContainerStarted","Data":"5586e8b86d0779e82a5c97dc6fd792c307c5fd7f2f4fc0796cba3eba28a9e57b"} Nov 02 23:21:40.427223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:40.427142 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" podStartSLOduration=2.427124224 podStartE2EDuration="2.427124224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:21:40.426607506 +0000 UTC m=+1693.833919339" watchObservedRunningTime="2026-11-02 23:21:40.427124224 +0000 UTC m=+1693.834436055" Nov 02 23:21:41.332328 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:41.332286 5672 generic.go:356] "Generic (PLEG): container finished" podID="89195d4e-ffa0-4231-95d8-5982b1aa34f0" containerID="47dff56f8bfafc357320da889739c681c35de59727a9b6ffb58ff8893ac9cd58" exitCode=0 Nov 02 23:21:41.332839 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:41.332419 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" event={"ID":"89195d4e-ffa0-4231-95d8-5982b1aa34f0","Type":"ContainerDied","Data":"47dff56f8bfafc357320da889739c681c35de59727a9b6ffb58ff8893ac9cd58"} Nov 02 23:21:41.872983 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:41.872647 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" podUID="224de738-298d-45a0-849e-389c7f9ee388" containerName="payload-processing" containerID="cri-o://cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364" gracePeriod=25 Nov 02 23:21:42.175259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.175234 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:21:42.280299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.280265 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/224de738-298d-45a0-849e-389c7f9ee388-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224de738-298d-45a0-849e-389c7f9ee388-plugins-config-volume\") pod \"224de738-298d-45a0-849e-389c7f9ee388\" (UID: \"224de738-298d-45a0-849e-389c7f9ee388\") " Nov 02 23:21:42.280591 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.280331 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/224de738-298d-45a0-849e-389c7f9ee388-kube-api-access-qd66f\" (UniqueName: \"kubernetes.io/projected/224de738-298d-45a0-849e-389c7f9ee388-kube-api-access-qd66f\") pod \"224de738-298d-45a0-849e-389c7f9ee388\" (UID: \"224de738-298d-45a0-849e-389c7f9ee388\") " Nov 02 23:21:42.280723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.280699 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224de738-298d-45a0-849e-389c7f9ee388-plugins-config-volume" pod "224de738-298d-45a0-849e-389c7f9ee388" (UID: "224de738-298d-45a0-849e-389c7f9ee388"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:42.280828 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.280812 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224de738-298d-45a0-849e-389c7f9ee388-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:42.282846 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.282819 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224de738-298d-45a0-849e-389c7f9ee388-kube-api-access-qd66f" pod "224de738-298d-45a0-849e-389c7f9ee388" (UID: "224de738-298d-45a0-849e-389c7f9ee388"). InnerVolumeSpecName "kube-api-access-qd66f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:42.339803 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.339767 5672 generic.go:356] "Generic (PLEG): container finished" podID="224de738-298d-45a0-849e-389c7f9ee388" containerID="cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364" exitCode=0 Nov 02 23:21:42.340247 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.339826 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" Nov 02 23:21:42.340247 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.339848 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" event={"ID":"224de738-298d-45a0-849e-389c7f9ee388","Type":"ContainerDied","Data":"cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364"} Nov 02 23:21:42.340247 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.339923 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx" event={"ID":"224de738-298d-45a0-849e-389c7f9ee388","Type":"ContainerDied","Data":"5c5396d117b085f1379a25ddbe5246fcb0b0c88bcf52defa0825b8b984cff701"} Nov 02 23:21:42.340247 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.339958 5672 scope.go:122] "RemoveContainer" containerID="cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364" Nov 02 23:21:42.351345 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.351324 5672 scope.go:122] "RemoveContainer" containerID="cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364" Nov 02 23:21:42.351616 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:21:42.351594 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364\": container with ID starting with cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364 not found: ID does not exist" containerID="cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364" Nov 02 23:21:42.351658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.351625 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364"} err="failed to get container status \"cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364\": rpc error: code = NotFound desc = could not find container \"cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364\": container with ID starting with cc2a8e07b9264411a0fc5b4b9d0ae2b44f3b3ffdf4c6b59645a5f543410b1364 not found: ID does not exist" Nov 02 23:21:42.382490 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.382453 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd66f\" (UniqueName: \"kubernetes.io/projected/224de738-298d-45a0-849e-389c7f9ee388-kube-api-access-qd66f\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:42.497243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.497211 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx"] Nov 02 23:21:42.497649 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.497631 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-68e067-79bc5db75f-2dpzx"] Nov 02 23:21:42.508693 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.508671 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:42.585034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.585009 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89195d4e-ffa0-4231-95d8-5982b1aa34f0-kube-api-access-2k2gm\" (UniqueName: \"kubernetes.io/projected/89195d4e-ffa0-4231-95d8-5982b1aa34f0-kube-api-access-2k2gm\") pod \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " Nov 02 23:21:42.585196 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.585055 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-xks-ca\") pod \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " Nov 02 23:21:42.585265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.585224 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-openshift-ca\") pod \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\" (UID: \"89195d4e-ffa0-4231-95d8-5982b1aa34f0\") " Nov 02 23:21:42.585601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.585576 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-openshift-ca" pod "89195d4e-ffa0-4231-95d8-5982b1aa34f0" (UID: "89195d4e-ffa0-4231-95d8-5982b1aa34f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:42.587096 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.587078 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-xks-ca" pod "89195d4e-ffa0-4231-95d8-5982b1aa34f0" (UID: "89195d4e-ffa0-4231-95d8-5982b1aa34f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:21:42.587427 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.587408 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89195d4e-ffa0-4231-95d8-5982b1aa34f0-kube-api-access-2k2gm" pod "89195d4e-ffa0-4231-95d8-5982b1aa34f0" (UID: "89195d4e-ffa0-4231-95d8-5982b1aa34f0"). InnerVolumeSpecName "kube-api-access-2k2gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:42.686751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.686679 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2gm\" (UniqueName: \"kubernetes.io/projected/89195d4e-ffa0-4231-95d8-5982b1aa34f0-kube-api-access-2k2gm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:42.686751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.686707 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:42.686751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.686717 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89195d4e-ffa0-4231-95d8-5982b1aa34f0-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:42.712871 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:42.712838 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="224de738-298d-45a0-849e-389c7f9ee388" path="/var/lib/kubelet/pods/224de738-298d-45a0-849e-389c7f9ee388/volumes" Nov 02 23:21:43.348013 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:43.347987 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" Nov 02 23:21:43.348418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:43.348031 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws" event={"ID":"89195d4e-ffa0-4231-95d8-5982b1aa34f0","Type":"ContainerDied","Data":"5586e8b86d0779e82a5c97dc6fd792c307c5fd7f2f4fc0796cba3eba28a9e57b"} Nov 02 23:21:43.348418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:43.348082 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5586e8b86d0779e82a5c97dc6fd792c307c5fd7f2f4fc0796cba3eba28a9e57b" Nov 02 23:21:46.318825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:46.318794 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w"] Nov 02 23:21:46.319255 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:46.318997 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" containerID="cri-o://5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46" gracePeriod=30 Nov 02 23:21:51.584168 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.584096 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:21:51.682366 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682314 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-kube-api-access-q9rds\" (UniqueName: \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-kube-api-access-q9rds\") pod \"ccc2dccd-0e58-471e-92c4-9981abf92255\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " Nov 02 23:21:51.682545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682393 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-certs\") pod \"ccc2dccd-0e58-471e-92c4-9981abf92255\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " Nov 02 23:21:51.682545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682517 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-data\") pod \"ccc2dccd-0e58-471e-92c4-9981abf92255\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " Nov 02 23:21:51.682670 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682599 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-token\") pod \"ccc2dccd-0e58-471e-92c4-9981abf92255\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " Nov 02 23:21:51.682670 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682660 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-credential-socket\") pod \"ccc2dccd-0e58-471e-92c4-9981abf92255\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " Nov 02 23:21:51.682784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682688 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-envoy\") pod \"ccc2dccd-0e58-471e-92c4-9981abf92255\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " Nov 02 23:21:51.682784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682715 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc2dccd-0e58-471e-92c4-9981abf92255-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2dccd-0e58-471e-92c4-9981abf92255-istiod-ca-cert\") pod \"ccc2dccd-0e58-471e-92c4-9981abf92255\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " Nov 02 23:21:51.682784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682725 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-certs" pod "ccc2dccd-0e58-471e-92c4-9981abf92255" (UID: "ccc2dccd-0e58-471e-92c4-9981abf92255"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:51.682784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682767 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-podinfo\") pod \"ccc2dccd-0e58-471e-92c4-9981abf92255\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " Nov 02 23:21:51.683052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682817 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-socket\") pod \"ccc2dccd-0e58-471e-92c4-9981abf92255\" (UID: \"ccc2dccd-0e58-471e-92c4-9981abf92255\") " Nov 02 23:21:51.683052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682761 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-data" pod "ccc2dccd-0e58-471e-92c4-9981abf92255" (UID: "ccc2dccd-0e58-471e-92c4-9981abf92255"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:51.683052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.682941 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-credential-socket" pod "ccc2dccd-0e58-471e-92c4-9981abf92255" (UID: "ccc2dccd-0e58-471e-92c4-9981abf92255"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:51.683267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.683134 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-socket" pod "ccc2dccd-0e58-471e-92c4-9981abf92255" (UID: "ccc2dccd-0e58-471e-92c4-9981abf92255"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:51.683267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.683251 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc2dccd-0e58-471e-92c4-9981abf92255-istiod-ca-cert" pod "ccc2dccd-0e58-471e-92c4-9981abf92255" (UID: "ccc2dccd-0e58-471e-92c4-9981abf92255"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:21:51.683470 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.683446 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:51.683556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.683477 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc2dccd-0e58-471e-92c4-9981abf92255-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:51.683556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.683492 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:51.683556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.683507 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:51.683556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.683533 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:51.684841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.684811 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-token" pod "ccc2dccd-0e58-471e-92c4-9981abf92255" (UID: "ccc2dccd-0e58-471e-92c4-9981abf92255"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:51.684841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.684827 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-envoy" pod "ccc2dccd-0e58-471e-92c4-9981abf92255" (UID: "ccc2dccd-0e58-471e-92c4-9981abf92255"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:21:51.685036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.685016 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-podinfo" pod "ccc2dccd-0e58-471e-92c4-9981abf92255" (UID: "ccc2dccd-0e58-471e-92c4-9981abf92255"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:21:51.685074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.685034 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-kube-api-access-q9rds" pod "ccc2dccd-0e58-471e-92c4-9981abf92255" (UID: "ccc2dccd-0e58-471e-92c4-9981abf92255"). InnerVolumeSpecName "kube-api-access-q9rds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:21:51.784770 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.784736 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:51.784770 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.784764 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:51.784770 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.784773 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc2dccd-0e58-471e-92c4-9981abf92255-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:51.785009 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:51.784784 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9rds\" (UniqueName: \"kubernetes.io/projected/ccc2dccd-0e58-471e-92c4-9981abf92255-kube-api-access-q9rds\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:21:52.420623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.420589 5672 generic.go:356] "Generic (PLEG): container finished" podID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerID="5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46" exitCode=0 Nov 02 23:21:52.420816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.420638 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" Nov 02 23:21:52.420816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.420672 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" event={"ID":"ccc2dccd-0e58-471e-92c4-9981abf92255","Type":"ContainerDied","Data":"5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46"} Nov 02 23:21:52.420816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.420730 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w" event={"ID":"ccc2dccd-0e58-471e-92c4-9981abf92255","Type":"ContainerDied","Data":"266a94578ffeb709fc4b66760ff0a51523f05e09aca9bbf0d891eee4f122588f"} Nov 02 23:21:52.420816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.420759 5672 scope.go:122] "RemoveContainer" containerID="5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46" Nov 02 23:21:52.433951 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.433930 5672 scope.go:122] "RemoveContainer" containerID="a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa" Nov 02 23:21:52.445729 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.445702 5672 scope.go:122] "RemoveContainer" containerID="5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46" Nov 02 23:21:52.446006 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:21:52.445984 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46\": container with ID starting with 5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46 not found: ID does not exist" containerID="5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46" Nov 02 23:21:52.446055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.446015 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46"} err="failed to get container status \"5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46\": rpc error: code = NotFound desc = could not find container \"5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46\": container with ID starting with 5d9615667dc92c58b6a58428acba1b724436fb2f1ed235315f5644a5ba75ee46 not found: ID does not exist" Nov 02 23:21:52.446055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.446033 5672 scope.go:122] "RemoveContainer" containerID="a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa" Nov 02 23:21:52.446337 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:21:52.446316 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa\": container with ID starting with a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa not found: ID does not exist" containerID="a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa" Nov 02 23:21:52.446392 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.446343 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa"} err="failed to get container status \"a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa\": rpc error: code = NotFound desc = could not find container \"a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa\": container with ID starting with a904aa227123afd449f35f8d6c4d0228bb7c5d39a9506bd8780321e0baf369fa not found: ID does not exist" Nov 02 23:21:52.625694 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.625663 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w"] Nov 02 23:21:52.672211 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.672117 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-79385a-openshift-default-8645744ccc-89b7w"] Nov 02 23:21:52.712539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:52.712510 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" path="/var/lib/kubelet/pods/ccc2dccd-0e58-471e-92c4-9981abf92255/volumes" Nov 02 23:21:54.045918 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:54.045884 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g"] Nov 02 23:21:54.103677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:54.103637 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74b7ec4-gpt6g"] Nov 02 23:21:54.713539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:21:54.713503 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd" path="/var/lib/kubelet/pods/b20bcb9e-48bf-4d27-b9a8-5dde3921d1fd/volumes" Nov 02 23:21:56.058317 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:21:56.058264 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:22:07.754702 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:07.754605 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:22:19.796577 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.796538 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7"] Nov 02 23:22:19.797095 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.797080 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" Nov 02 23:22:19.797138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.797096 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" Nov 02 23:22:19.797138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.797110 5672 cpu_manager.go:414] "containerMap: removing container" podUID="224de738-298d-45a0-849e-389c7f9ee388" containerName="payload-processing" Nov 02 23:22:19.797138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.797116 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224de738-298d-45a0-849e-389c7f9ee388" containerName="payload-processing" Nov 02 23:22:19.797297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.797140 5672 cpu_manager.go:414] "containerMap: removing container" podUID="89195d4e-ffa0-4231-95d8-5982b1aa34f0" containerName="revoke-keys" Nov 02 23:22:19.797297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.797156 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89195d4e-ffa0-4231-95d8-5982b1aa34f0" containerName="revoke-keys" Nov 02 23:22:19.797297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.797234 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="89195d4e-ffa0-4231-95d8-5982b1aa34f0" containerName="revoke-keys" Nov 02 23:22:19.797297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.797248 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" Nov 02 23:22:19.797297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.797256 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" Nov 02 23:22:19.797297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.797262 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="224de738-298d-45a0-849e-389c7f9ee388" containerName="payload-processing" Nov 02 23:22:19.800539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.800517 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:22:19.856016 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.855981 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7"] Nov 02 23:22:19.897461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.897427 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c0064f-dockercfg-smrss\"" Nov 02 23:22:19.897461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.897448 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c0064f\"" Nov 02 23:22:19.942171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.927278 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e48b30-2e0c-4996-8db6-613c2a068866-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7\" (UID: \"96e48b30-2e0c-4996-8db6-613c2a068866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:22:19.966919 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.966883 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6qts\" (UniqueName: \"kubernetes.io/projected/96e48b30-2e0c-4996-8db6-613c2a068866-kube-api-access-l6qts\") pod \"payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7\" (UID: \"96e48b30-2e0c-4996-8db6-613c2a068866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:22:19.989137 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.989101 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9"] Nov 02 23:22:19.991426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.991370 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" Nov 02 23:22:19.999450 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:19.999425 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc2dccd-0e58-471e-92c4-9981abf92255" containerName="istio-proxy" Nov 02 23:22:20.003681 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.003656 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:22:20.026351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.026320 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9"] Nov 02 23:22:20.079512 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.079413 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bcaf02-f310-4d31-b36d-1765952304e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9\" (UID: \"47bcaf02-f310-4d31-b36d-1765952304e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:22:20.079688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.079596 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb45l\" (UniqueName: \"kubernetes.io/projected/47bcaf02-f310-4d31-b36d-1765952304e7-kube-api-access-zb45l\") pod \"payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9\" (UID: \"47bcaf02-f310-4d31-b36d-1765952304e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:22:20.079761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.079701 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e48b30-2e0c-4996-8db6-613c2a068866-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7\" (UID: \"96e48b30-2e0c-4996-8db6-613c2a068866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:22:20.079761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.079734 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qts\" (UniqueName: \"kubernetes.io/projected/96e48b30-2e0c-4996-8db6-613c2a068866-kube-api-access-l6qts\") pod \"payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7\" (UID: \"96e48b30-2e0c-4996-8db6-613c2a068866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:22:20.081103 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.081076 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e48b30-2e0c-4996-8db6-613c2a068866-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7\" (UID: \"96e48b30-2e0c-4996-8db6-613c2a068866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:22:20.133242 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.133210 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qts\" (UniqueName: \"kubernetes.io/projected/96e48b30-2e0c-4996-8db6-613c2a068866-kube-api-access-l6qts\") pod \"payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7\" (UID: \"96e48b30-2e0c-4996-8db6-613c2a068866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:22:20.180838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.180793 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bcaf02-f310-4d31-b36d-1765952304e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9\" (UID: \"47bcaf02-f310-4d31-b36d-1765952304e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:22:20.181034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.180919 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb45l\" (UniqueName: \"kubernetes.io/projected/47bcaf02-f310-4d31-b36d-1765952304e7-kube-api-access-zb45l\") pod \"payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9\" (UID: \"47bcaf02-f310-4d31-b36d-1765952304e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:22:20.181571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.181551 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bcaf02-f310-4d31-b36d-1765952304e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9\" (UID: \"47bcaf02-f310-4d31-b36d-1765952304e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:22:20.234025 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.233986 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb45l\" (UniqueName: \"kubernetes.io/projected/47bcaf02-f310-4d31-b36d-1765952304e7-kube-api-access-zb45l\") pod \"payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9\" (UID: \"47bcaf02-f310-4d31-b36d-1765952304e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:22:20.319578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.319547 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:22:20.422366 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.415556 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:22:20.698539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.698507 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9"] Nov 02 23:22:20.704606 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:22:20.704578 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bcaf02_f310_4d31_b36d_1765952304e7.slice/crio-387840944bf558a71f7162370da20d06e1092da1e6f4d37fb459f1078e2241e0 WatchSource:0}: Error finding container 387840944bf558a71f7162370da20d06e1092da1e6f4d37fb459f1078e2241e0: Status 404 returned error can't find the container with id 387840944bf558a71f7162370da20d06e1092da1e6f4d37fb459f1078e2241e0 Nov 02 23:22:20.805142 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:20.805116 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7"] Nov 02 23:22:20.808515 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:22:20.808477 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e48b30_2e0c_4996_8db6_613c2a068866.slice/crio-e6ff181f5b3e2849643c402538e15162e7a3b7e5d1f4f4048ba6ab594231e4a2 WatchSource:0}: Error finding container e6ff181f5b3e2849643c402538e15162e7a3b7e5d1f4f4048ba6ab594231e4a2: Status 404 returned error can't find the container with id e6ff181f5b3e2849643c402538e15162e7a3b7e5d1f4f4048ba6ab594231e4a2 Nov 02 23:22:21.653862 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:21.653826 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" event={"ID":"96e48b30-2e0c-4996-8db6-613c2a068866","Type":"ContainerStarted","Data":"ee9ffb55d940aa0a2d4ff40a1620fdf0a3d0619d9375c192518eb825632872df"} Nov 02 23:22:21.654075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:21.653878 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" event={"ID":"96e48b30-2e0c-4996-8db6-613c2a068866","Type":"ContainerStarted","Data":"e6ff181f5b3e2849643c402538e15162e7a3b7e5d1f4f4048ba6ab594231e4a2"} Nov 02 23:22:21.654075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:21.653904 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:22:21.655641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:21.655617 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" event={"ID":"47bcaf02-f310-4d31-b36d-1765952304e7","Type":"ContainerStarted","Data":"0a2f0bd020ea4d83356f3381f019c30034f1f5a85c129c607d97d0e9bcb4478d"} Nov 02 23:22:21.655764 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:21.655656 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" event={"ID":"47bcaf02-f310-4d31-b36d-1765952304e7","Type":"ContainerStarted","Data":"387840944bf558a71f7162370da20d06e1092da1e6f4d37fb459f1078e2241e0"} Nov 02 23:22:21.655764 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:21.655677 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:22:21.762306 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:21.762234 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" podStartSLOduration=2.762217471 podStartE2EDuration="2.762217471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:21.759570811 +0000 UTC m=+1735.166882655" watchObservedRunningTime="2026-11-02 23:22:21.762217471 +0000 UTC m=+1735.169529301" Nov 02 23:22:21.867687 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:21.867636 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" podStartSLOduration=2.867622643 podStartE2EDuration="2.867622643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:21.865034413 +0000 UTC m=+1735.272346243" watchObservedRunningTime="2026-11-02 23:22:21.867622643 +0000 UTC m=+1735.274934472" Nov 02 23:22:23.043636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:23.043602 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq"] Nov 02 23:22:23.090024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:23.089988 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dbdeeaf-gwkwq"] Nov 02 23:22:24.714260 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:24.714228 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08ed4a3-d627-4111-9a3e-372d01899e99" path="/var/lib/kubelet/pods/c08ed4a3-d627-4111-9a3e-372d01899e99/volumes" Nov 02 23:22:26.021549 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:22:26.021513 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:22:27.826421 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:27.826388 5672 scope.go:122] "RemoveContainer" containerID="ef2bd1e0d377525373f63140065854516322151b1154a506623d1b19e7b34caa" Nov 02 23:22:32.664498 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:32.664469 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:22:32.664904 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:32.664535 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:22:38.863142 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:38.863110 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws"] Nov 02 23:22:38.900302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:38.900273 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e953257b-nmgws"] Nov 02 23:22:39.107434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:39.107399 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627"] Nov 02 23:22:39.107968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:39.107911 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" podUID="28ef579f-a819-4988-ab22-eafd9da5e6b7" containerName="istio-proxy" containerID="cri-o://f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60" gracePeriod=30 Nov 02 23:22:39.127437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:39.127347 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn"] Nov 02 23:22:39.127873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:39.127842 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" podUID="21f92e90-6561-40df-89d2-754922005318" containerName="istio-proxy" containerID="cri-o://2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7" gracePeriod=30 Nov 02 23:22:39.140952 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:39.140926 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg"] Nov 02 23:22:39.141187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:39.141165 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" podUID="a206e651-2148-4971-9c30-8ed88959a6b4" containerName="istio-proxy" containerID="cri-o://b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399" gracePeriod=30 Nov 02 23:22:40.714732 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:40.714692 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89195d4e-ffa0-4231-95d8-5982b1aa34f0" path="/var/lib/kubelet/pods/89195d4e-ffa0-4231-95d8-5982b1aa34f0/volumes" Nov 02 23:22:44.431407 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.431374 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:22:44.499680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.499655 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:22:44.502857 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.502839 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:22:44.507379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507357 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-envoy\") pod \"28ef579f-a819-4988-ab22-eafd9da5e6b7\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " Nov 02 23:22:44.507482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507472 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f92e90-6561-40df-89d2-754922005318-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f92e90-6561-40df-89d2-754922005318-istiod-ca-cert\") pod \"21f92e90-6561-40df-89d2-754922005318\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " Nov 02 23:22:44.507547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507499 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-kube-api-access-qp9wg\" (UniqueName: \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-kube-api-access-qp9wg\") pod \"a206e651-2148-4971-9c30-8ed88959a6b4\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " Nov 02 23:22:44.507547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507538 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ef579f-a819-4988-ab22-eafd9da5e6b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ef579f-a819-4988-ab22-eafd9da5e6b7-istiod-ca-cert\") pod \"28ef579f-a819-4988-ab22-eafd9da5e6b7\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " Nov 02 23:22:44.507658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507562 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-certs\") pod \"a206e651-2148-4971-9c30-8ed88959a6b4\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " Nov 02 23:22:44.507658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507613 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-envoy\") pod \"a206e651-2148-4971-9c30-8ed88959a6b4\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " Nov 02 23:22:44.507658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507642 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-istio-token\" (UniqueName: \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-istio-token\") pod \"21f92e90-6561-40df-89d2-754922005318\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " Nov 02 23:22:44.507780 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507680 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-certs\") pod \"21f92e90-6561-40df-89d2-754922005318\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " Nov 02 23:22:44.507780 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507723 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a206e651-2148-4971-9c30-8ed88959a6b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a206e651-2148-4971-9c30-8ed88959a6b4-istio-podinfo\") pod \"a206e651-2148-4971-9c30-8ed88959a6b4\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " Nov 02 23:22:44.507844 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507783 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21f92e90-6561-40df-89d2-754922005318-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f92e90-6561-40df-89d2-754922005318-istio-podinfo\") pod \"21f92e90-6561-40df-89d2-754922005318\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " Nov 02 23:22:44.507844 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507817 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-kube-api-access-jr4pg\" (UniqueName: \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-kube-api-access-jr4pg\") pod \"21f92e90-6561-40df-89d2-754922005318\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " Nov 02 23:22:44.507912 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507878 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-data\") pod \"21f92e90-6561-40df-89d2-754922005318\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " Nov 02 23:22:44.507977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507912 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-certs" pod "a206e651-2148-4971-9c30-8ed88959a6b4" (UID: "a206e651-2148-4971-9c30-8ed88959a6b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.507977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507921 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-token\" (UniqueName: \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-token\") pod \"28ef579f-a819-4988-ab22-eafd9da5e6b7\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " Nov 02 23:22:44.507977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507924 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f92e90-6561-40df-89d2-754922005318-istiod-ca-cert" pod "21f92e90-6561-40df-89d2-754922005318" (UID: "21f92e90-6561-40df-89d2-754922005318"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:44.507977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507945 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ef579f-a819-4988-ab22-eafd9da5e6b7-istiod-ca-cert" pod "28ef579f-a819-4988-ab22-eafd9da5e6b7" (UID: "28ef579f-a819-4988-ab22-eafd9da5e6b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:44.507977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.507950 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-socket\") pod \"21f92e90-6561-40df-89d2-754922005318\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " Nov 02 23:22:44.508245 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508005 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-credential-socket\") pod \"28ef579f-a819-4988-ab22-eafd9da5e6b7\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " Nov 02 23:22:44.508245 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508039 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-certs\") pod \"28ef579f-a819-4988-ab22-eafd9da5e6b7\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " Nov 02 23:22:44.508245 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508095 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-podinfo\") pod \"28ef579f-a819-4988-ab22-eafd9da5e6b7\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " Nov 02 23:22:44.508245 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508191 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-socket" pod "21f92e90-6561-40df-89d2-754922005318" (UID: "21f92e90-6561-40df-89d2-754922005318"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.508245 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508225 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-data\") pod \"28ef579f-a819-4988-ab22-eafd9da5e6b7\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " Nov 02 23:22:44.508513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508257 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a206e651-2148-4971-9c30-8ed88959a6b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a206e651-2148-4971-9c30-8ed88959a6b4-istiod-ca-cert\") pod \"a206e651-2148-4971-9c30-8ed88959a6b4\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " Nov 02 23:22:44.508513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508302 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-socket\") pod \"28ef579f-a819-4988-ab22-eafd9da5e6b7\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " Nov 02 23:22:44.508513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508351 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-data\") pod \"a206e651-2148-4971-9c30-8ed88959a6b4\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " Nov 02 23:22:44.508513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508381 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-envoy\") pod \"21f92e90-6561-40df-89d2-754922005318\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " Nov 02 23:22:44.508513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508451 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-socket\") pod \"a206e651-2148-4971-9c30-8ed88959a6b4\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " Nov 02 23:22:44.508513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508501 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-istio-token\" (UniqueName: \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-istio-token\") pod \"a206e651-2148-4971-9c30-8ed88959a6b4\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " Nov 02 23:22:44.508836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508546 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-credential-socket\") pod \"a206e651-2148-4971-9c30-8ed88959a6b4\" (UID: \"a206e651-2148-4971-9c30-8ed88959a6b4\") " Nov 02 23:22:44.508836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508579 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-kube-api-access-h6nzl\" (UniqueName: \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-kube-api-access-h6nzl\") pod \"28ef579f-a819-4988-ab22-eafd9da5e6b7\" (UID: \"28ef579f-a819-4988-ab22-eafd9da5e6b7\") " Nov 02 23:22:44.508836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508620 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-certs" pod "21f92e90-6561-40df-89d2-754922005318" (UID: "21f92e90-6561-40df-89d2-754922005318"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.508836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508632 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-credential-socket\") pod \"21f92e90-6561-40df-89d2-754922005318\" (UID: \"21f92e90-6561-40df-89d2-754922005318\") " Nov 02 23:22:44.508836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.508640 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-data" pod "21f92e90-6561-40df-89d2-754922005318" (UID: "21f92e90-6561-40df-89d2-754922005318"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509255 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-socket" pod "28ef579f-a819-4988-ab22-eafd9da5e6b7" (UID: "28ef579f-a819-4988-ab22-eafd9da5e6b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509439 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-data" pod "28ef579f-a819-4988-ab22-eafd9da5e6b7" (UID: "28ef579f-a819-4988-ab22-eafd9da5e6b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509444 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509502 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509520 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509537 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f92e90-6561-40df-89d2-754922005318-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509552 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ef579f-a819-4988-ab22-eafd9da5e6b7-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509567 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509582 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509622 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-socket" pod "a206e651-2148-4971-9c30-8ed88959a6b4" (UID: "a206e651-2148-4971-9c30-8ed88959a6b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509668 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-data" pod "a206e651-2148-4971-9c30-8ed88959a6b4" (UID: "a206e651-2148-4971-9c30-8ed88959a6b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509680 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a206e651-2148-4971-9c30-8ed88959a6b4-istiod-ca-cert" pod "a206e651-2148-4971-9c30-8ed88959a6b4" (UID: "a206e651-2148-4971-9c30-8ed88959a6b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509880 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-credential-socket" pod "a206e651-2148-4971-9c30-8ed88959a6b4" (UID: "a206e651-2148-4971-9c30-8ed88959a6b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.510253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.509917 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-credential-socket" pod "28ef579f-a819-4988-ab22-eafd9da5e6b7" (UID: "28ef579f-a819-4988-ab22-eafd9da5e6b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.511214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.510259 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-credential-socket" pod "21f92e90-6561-40df-89d2-754922005318" (UID: "21f92e90-6561-40df-89d2-754922005318"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.511214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.510338 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-envoy" pod "28ef579f-a819-4988-ab22-eafd9da5e6b7" (UID: "28ef579f-a819-4988-ab22-eafd9da5e6b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.511214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.510388 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-envoy" pod "a206e651-2148-4971-9c30-8ed88959a6b4" (UID: "a206e651-2148-4971-9c30-8ed88959a6b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.511378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.511237 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-certs" pod "28ef579f-a819-4988-ab22-eafd9da5e6b7" (UID: "28ef579f-a819-4988-ab22-eafd9da5e6b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.512735 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.512690 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-istio-token" pod "21f92e90-6561-40df-89d2-754922005318" (UID: "21f92e90-6561-40df-89d2-754922005318"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.512838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.512781 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-podinfo" pod "28ef579f-a819-4988-ab22-eafd9da5e6b7" (UID: "28ef579f-a819-4988-ab22-eafd9da5e6b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:44.512903 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.512853 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-istio-token" pod "a206e651-2148-4971-9c30-8ed88959a6b4" (UID: "a206e651-2148-4971-9c30-8ed88959a6b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.513010 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.512991 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a206e651-2148-4971-9c30-8ed88959a6b4-istio-podinfo" pod "a206e651-2148-4971-9c30-8ed88959a6b4" (UID: "a206e651-2148-4971-9c30-8ed88959a6b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:44.513172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.513023 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21f92e90-6561-40df-89d2-754922005318-istio-podinfo" pod "21f92e90-6561-40df-89d2-754922005318" (UID: "21f92e90-6561-40df-89d2-754922005318"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:22:44.513172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.513130 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-kube-api-access-qp9wg" pod "a206e651-2148-4971-9c30-8ed88959a6b4" (UID: "a206e651-2148-4971-9c30-8ed88959a6b4"). InnerVolumeSpecName "kube-api-access-qp9wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.513331 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.513201 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-kube-api-access-h6nzl" pod "28ef579f-a819-4988-ab22-eafd9da5e6b7" (UID: "28ef579f-a819-4988-ab22-eafd9da5e6b7"). InnerVolumeSpecName "kube-api-access-h6nzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.513405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.513385 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-envoy" pod "21f92e90-6561-40df-89d2-754922005318" (UID: "21f92e90-6561-40df-89d2-754922005318"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:22:44.513526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.513509 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-kube-api-access-jr4pg" pod "21f92e90-6561-40df-89d2-754922005318" (UID: "21f92e90-6561-40df-89d2-754922005318"). InnerVolumeSpecName "kube-api-access-jr4pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.514015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.513999 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-token" pod "28ef579f-a819-4988-ab22-eafd9da5e6b7" (UID: "28ef579f-a819-4988-ab22-eafd9da5e6b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:22:44.611419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611382 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp9wg\" (UniqueName: \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-kube-api-access-qp9wg\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611409 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611418 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611428 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a206e651-2148-4971-9c30-8ed88959a6b4-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611438 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f92e90-6561-40df-89d2-754922005318-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611448 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr4pg\" (UniqueName: \"kubernetes.io/projected/21f92e90-6561-40df-89d2-754922005318-kube-api-access-jr4pg\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611457 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611465 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611473 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611481 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611490 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611498 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a206e651-2148-4971-9c30-8ed88959a6b4-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611506 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611514 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611522 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611530 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a206e651-2148-4971-9c30-8ed88959a6b4-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611538 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a206e651-2148-4971-9c30-8ed88959a6b4-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611546 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6nzl\" (UniqueName: \"kubernetes.io/projected/28ef579f-a819-4988-ab22-eafd9da5e6b7-kube-api-access-h6nzl\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611554 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f92e90-6561-40df-89d2-754922005318-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.611779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.611562 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ef579f-a819-4988-ab22-eafd9da5e6b7-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:22:44.832079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.831851 5672 generic.go:356] "Generic (PLEG): container finished" podID="28ef579f-a819-4988-ab22-eafd9da5e6b7" containerID="f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60" exitCode=0 Nov 02 23:22:44.833200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.832987 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" event={"ID":"28ef579f-a819-4988-ab22-eafd9da5e6b7","Type":"ContainerDied","Data":"f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60"} Nov 02 23:22:44.833200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.833095 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" event={"ID":"28ef579f-a819-4988-ab22-eafd9da5e6b7","Type":"ContainerDied","Data":"8adb283fb28315e4ec018cdf9835ff847b9aac3b4ddadb48d2c85e9d3d2f1317"} Nov 02 23:22:44.833200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.833135 5672 scope.go:122] "RemoveContainer" containerID="f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60" Nov 02 23:22:44.835344 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.834397 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627" Nov 02 23:22:44.838480 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.838452 5672 generic.go:356] "Generic (PLEG): container finished" podID="a206e651-2148-4971-9c30-8ed88959a6b4" containerID="b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399" exitCode=0 Nov 02 23:22:44.838591 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.838564 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" event={"ID":"a206e651-2148-4971-9c30-8ed88959a6b4","Type":"ContainerDied","Data":"b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399"} Nov 02 23:22:44.838661 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.838605 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" event={"ID":"a206e651-2148-4971-9c30-8ed88959a6b4","Type":"ContainerDied","Data":"e745381cfb598ad2131a51be9ef502d484d7a04e40909e35d2a8f486c8af11a2"} Nov 02 23:22:44.838722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.838680 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg" Nov 02 23:22:44.842515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.842486 5672 generic.go:356] "Generic (PLEG): container finished" podID="21f92e90-6561-40df-89d2-754922005318" containerID="2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7" exitCode=0 Nov 02 23:22:44.842614 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.842546 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" event={"ID":"21f92e90-6561-40df-89d2-754922005318","Type":"ContainerDied","Data":"2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7"} Nov 02 23:22:44.842614 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.842585 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" event={"ID":"21f92e90-6561-40df-89d2-754922005318","Type":"ContainerDied","Data":"ddeffdddcc483ba98c893324efd7915286a62fbde9cef1dc02d4a7682f5cee41"} Nov 02 23:22:44.843133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.843112 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn" Nov 02 23:22:44.871220 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.871196 5672 scope.go:122] "RemoveContainer" containerID="f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60" Nov 02 23:22:44.871543 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:22:44.871519 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60\": container with ID starting with f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60 not found: ID does not exist" containerID="f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60" Nov 02 23:22:44.871626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.871555 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60"} err="failed to get container status \"f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60\": rpc error: code = NotFound desc = could not find container \"f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60\": container with ID starting with f0f56f43fbeddcaada0d5ab60a7ed82d650e5860fd114e2db852e21617c5ca60 not found: ID does not exist" Nov 02 23:22:44.871626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.871580 5672 scope.go:122] "RemoveContainer" containerID="b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399" Nov 02 23:22:44.881125 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.881095 5672 scope.go:122] "RemoveContainer" containerID="b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399" Nov 02 23:22:44.881433 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:22:44.881412 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399\": container with ID starting with b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399 not found: ID does not exist" containerID="b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399" Nov 02 23:22:44.881498 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.881441 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399"} err="failed to get container status \"b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399\": rpc error: code = NotFound desc = could not find container \"b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399\": container with ID starting with b147b5e9b5c61d67863909aaefc8971a80911a9a7d4bbcede6ebd3cd0a1b9399 not found: ID does not exist" Nov 02 23:22:44.881498 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.881458 5672 scope.go:122] "RemoveContainer" containerID="2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7" Nov 02 23:22:44.890701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.890678 5672 scope.go:122] "RemoveContainer" containerID="2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7" Nov 02 23:22:44.891046 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:22:44.891026 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7\": container with ID starting with 2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7 not found: ID does not exist" containerID="2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7" Nov 02 23:22:44.891109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.891053 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7"} err="failed to get container status \"2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7\": rpc error: code = NotFound desc = could not find container \"2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7\": container with ID starting with 2610c858a9b009b1f59d921b669f09f6c38ec61d3ddc44be17ff8be860f1baf7 not found: ID does not exist" Nov 02 23:22:44.990849 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.990769 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn"] Nov 02 23:22:44.998482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:44.998455 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-6klwn"] Nov 02 23:22:45.100494 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:45.100464 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627"] Nov 02 23:22:45.103573 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:45.103546 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-25627"] Nov 02 23:22:45.205324 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:45.205285 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg"] Nov 02 23:22:45.208935 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:45.208911 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-pfbfg"] Nov 02 23:22:46.712561 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:46.712528 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f92e90-6561-40df-89d2-754922005318" path="/var/lib/kubelet/pods/21f92e90-6561-40df-89d2-754922005318/volumes" Nov 02 23:22:46.713031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:46.713017 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ef579f-a819-4988-ab22-eafd9da5e6b7" path="/var/lib/kubelet/pods/28ef579f-a819-4988-ab22-eafd9da5e6b7/volumes" Nov 02 23:22:46.713469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:46.713456 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a206e651-2148-4971-9c30-8ed88959a6b4" path="/var/lib/kubelet/pods/a206e651-2148-4971-9c30-8ed88959a6b4/volumes" Nov 02 23:22:49.171096 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:49.171057 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:22:51.106252 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.106208 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp"] Nov 02 23:22:51.106880 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.106853 5672 cpu_manager.go:414] "containerMap: removing container" podUID="21f92e90-6561-40df-89d2-754922005318" containerName="istio-proxy" Nov 02 23:22:51.106880 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.106881 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f92e90-6561-40df-89d2-754922005318" containerName="istio-proxy" Nov 02 23:22:51.107093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.106898 5672 cpu_manager.go:414] "containerMap: removing container" podUID="28ef579f-a819-4988-ab22-eafd9da5e6b7" containerName="istio-proxy" Nov 02 23:22:51.107093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.106905 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ef579f-a819-4988-ab22-eafd9da5e6b7" containerName="istio-proxy" Nov 02 23:22:51.107093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.106948 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a206e651-2148-4971-9c30-8ed88959a6b4" containerName="istio-proxy" Nov 02 23:22:51.107093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.106956 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a206e651-2148-4971-9c30-8ed88959a6b4" containerName="istio-proxy" Nov 02 23:22:51.107093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.107080 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f92e90-6561-40df-89d2-754922005318" containerName="istio-proxy" Nov 02 23:22:51.107093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.107094 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ef579f-a819-4988-ab22-eafd9da5e6b7" containerName="istio-proxy" Nov 02 23:22:51.107856 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.107105 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a206e651-2148-4971-9c30-8ed88959a6b4" containerName="istio-proxy" Nov 02 23:22:51.111974 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.111952 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:22:51.181803 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.173905 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp"] Nov 02 23:22:51.264250 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.260301 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b57963\"" Nov 02 23:22:51.264250 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.260467 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b57963-dockercfg-v22j8\"" Nov 02 23:22:51.273955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.273917 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c267ca54-0618-4561-9d5f-95c3c7cb655b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp\" (UID: \"c267ca54-0618-4561-9d5f-95c3c7cb655b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:22:51.284404 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.284366 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbtr\" (UniqueName: \"kubernetes.io/projected/c267ca54-0618-4561-9d5f-95c3c7cb655b-kube-api-access-rpbtr\") pod \"payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp\" (UID: \"c267ca54-0618-4561-9d5f-95c3c7cb655b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:22:51.307504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.307443 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh"] Nov 02 23:22:51.320017 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.319987 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:22:51.345123 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.345008 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh"] Nov 02 23:22:51.396940 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.396850 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ddc0a44-811e-40dc-ac64-57a75bf439be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh\" (UID: \"2ddc0a44-811e-40dc-ac64-57a75bf439be\") " pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:22:51.397098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.396995 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c267ca54-0618-4561-9d5f-95c3c7cb655b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp\" (UID: \"c267ca54-0618-4561-9d5f-95c3c7cb655b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:22:51.397098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.397081 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r45fx\" (UniqueName: \"kubernetes.io/projected/2ddc0a44-811e-40dc-ac64-57a75bf439be-kube-api-access-r45fx\") pod \"payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh\" (UID: \"2ddc0a44-811e-40dc-ac64-57a75bf439be\") " pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:22:51.397208 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.397130 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbtr\" (UniqueName: \"kubernetes.io/projected/c267ca54-0618-4561-9d5f-95c3c7cb655b-kube-api-access-rpbtr\") pod \"payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp\" (UID: \"c267ca54-0618-4561-9d5f-95c3c7cb655b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:22:51.397845 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.397826 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c267ca54-0618-4561-9d5f-95c3c7cb655b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp\" (UID: \"c267ca54-0618-4561-9d5f-95c3c7cb655b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:22:51.450297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.450255 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbtr\" (UniqueName: \"kubernetes.io/projected/c267ca54-0618-4561-9d5f-95c3c7cb655b-kube-api-access-rpbtr\") pod \"payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp\" (UID: \"c267ca54-0618-4561-9d5f-95c3c7cb655b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:22:51.498811 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.498769 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r45fx\" (UniqueName: \"kubernetes.io/projected/2ddc0a44-811e-40dc-ac64-57a75bf439be-kube-api-access-r45fx\") pod \"payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh\" (UID: \"2ddc0a44-811e-40dc-ac64-57a75bf439be\") " pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:22:51.499012 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.498953 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ddc0a44-811e-40dc-ac64-57a75bf439be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh\" (UID: \"2ddc0a44-811e-40dc-ac64-57a75bf439be\") " pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:22:51.499927 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.499903 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ddc0a44-811e-40dc-ac64-57a75bf439be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh\" (UID: \"2ddc0a44-811e-40dc-ac64-57a75bf439be\") " pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:22:51.552236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.552206 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r45fx\" (UniqueName: \"kubernetes.io/projected/2ddc0a44-811e-40dc-ac64-57a75bf439be-kube-api-access-r45fx\") pod \"payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh\" (UID: \"2ddc0a44-811e-40dc-ac64-57a75bf439be\") " pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:22:51.646274 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.646238 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:22:51.727074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:51.726956 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:22:52.038695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:52.038657 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh"] Nov 02 23:22:52.040744 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:22:52.040718 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddc0a44_811e_40dc_ac64_57a75bf439be.slice/crio-b2417fd747689c46b174c1c118ffc85b086c5cc971eb3576e9a12065d6b18216 WatchSource:0}: Error finding container b2417fd747689c46b174c1c118ffc85b086c5cc971eb3576e9a12065d6b18216: Status 404 returned error can't find the container with id b2417fd747689c46b174c1c118ffc85b086c5cc971eb3576e9a12065d6b18216 Nov 02 23:22:52.148449 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:52.148419 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp"] Nov 02 23:22:52.150395 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:22:52.150365 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc267ca54_0618_4561_9d5f_95c3c7cb655b.slice/crio-ed5f8b27ee2433a171af1d75f2033eec8cb824c16ff6cdf7225ccda60b65446b WatchSource:0}: Error finding container ed5f8b27ee2433a171af1d75f2033eec8cb824c16ff6cdf7225ccda60b65446b: Status 404 returned error can't find the container with id ed5f8b27ee2433a171af1d75f2033eec8cb824c16ff6cdf7225ccda60b65446b Nov 02 23:22:52.908598 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:52.908560 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" event={"ID":"c267ca54-0618-4561-9d5f-95c3c7cb655b","Type":"ContainerStarted","Data":"62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42"} Nov 02 23:22:52.908798 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:52.908660 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" event={"ID":"c267ca54-0618-4561-9d5f-95c3c7cb655b","Type":"ContainerStarted","Data":"ed5f8b27ee2433a171af1d75f2033eec8cb824c16ff6cdf7225ccda60b65446b"} Nov 02 23:22:52.908857 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:52.908794 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:22:52.910192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:52.910134 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" event={"ID":"2ddc0a44-811e-40dc-ac64-57a75bf439be","Type":"ContainerStarted","Data":"ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876"} Nov 02 23:22:52.910319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:52.910202 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" event={"ID":"2ddc0a44-811e-40dc-ac64-57a75bf439be","Type":"ContainerStarted","Data":"b2417fd747689c46b174c1c118ffc85b086c5cc971eb3576e9a12065d6b18216"} Nov 02 23:22:52.910319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:52.910223 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:22:53.018472 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:53.018420 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" podStartSLOduration=2.018407429 podStartE2EDuration="2.018407429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:53.01655396 +0000 UTC m=+1766.423865789" watchObservedRunningTime="2026-11-02 23:22:53.018407429 +0000 UTC m=+1766.425719286" Nov 02 23:22:53.124241 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:53.124172 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" podStartSLOduration=2.12413327 podStartE2EDuration="2.12413327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:53.122818117 +0000 UTC m=+1766.530129951" watchObservedRunningTime="2026-11-02 23:22:53.12413327 +0000 UTC m=+1766.531445100" Nov 02 23:22:56.021052 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:22:56.021011 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:22:57.906950 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:57.906915 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql"] Nov 02 23:22:57.911168 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:57.911127 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:22:57.982267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:57.981370 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql"] Nov 02 23:22:57.997171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:57.992883 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb948\" (UniqueName: \"kubernetes.io/projected/f80a5bc1-7bcc-4839-84c7-28253d42a95d-kube-api-access-wb948\") pod \"payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql\" (UID: \"f80a5bc1-7bcc-4839-84c7-28253d42a95d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:22:57.997171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:57.992959 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80a5bc1-7bcc-4839-84c7-28253d42a95d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql\" (UID: \"f80a5bc1-7bcc-4839-84c7-28253d42a95d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:22:58.014445 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:58.014411 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9f5790\"" Nov 02 23:22:58.014688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:58.014525 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9f5790-dockercfg-wxnn6\"" Nov 02 23:22:58.094626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:58.094581 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb948\" (UniqueName: \"kubernetes.io/projected/f80a5bc1-7bcc-4839-84c7-28253d42a95d-kube-api-access-wb948\") pod \"payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql\" (UID: \"f80a5bc1-7bcc-4839-84c7-28253d42a95d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:22:58.094832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:58.094665 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80a5bc1-7bcc-4839-84c7-28253d42a95d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql\" (UID: \"f80a5bc1-7bcc-4839-84c7-28253d42a95d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:22:58.095531 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:58.095484 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80a5bc1-7bcc-4839-84c7-28253d42a95d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql\" (UID: \"f80a5bc1-7bcc-4839-84c7-28253d42a95d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:22:58.149368 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:58.149336 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb948\" (UniqueName: \"kubernetes.io/projected/f80a5bc1-7bcc-4839-84c7-28253d42a95d-kube-api-access-wb948\") pod \"payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql\" (UID: \"f80a5bc1-7bcc-4839-84c7-28253d42a95d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:22:58.225272 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:58.225235 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:22:58.600826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:58.600795 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql"] Nov 02 23:22:58.602397 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:22:58.602371 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80a5bc1_7bcc_4839_84c7_28253d42a95d.slice/crio-a1aa49622953a20e19d79ea848bd59ebd32bb1d9511f14ff2df1b8bb62eae788 WatchSource:0}: Error finding container a1aa49622953a20e19d79ea848bd59ebd32bb1d9511f14ff2df1b8bb62eae788: Status 404 returned error can't find the container with id a1aa49622953a20e19d79ea848bd59ebd32bb1d9511f14ff2df1b8bb62eae788 Nov 02 23:22:58.968453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:58.968424 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" event={"ID":"f80a5bc1-7bcc-4839-84c7-28253d42a95d","Type":"ContainerStarted","Data":"c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35"} Nov 02 23:22:58.968933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:58.968480 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" event={"ID":"f80a5bc1-7bcc-4839-84c7-28253d42a95d","Type":"ContainerStarted","Data":"a1aa49622953a20e19d79ea848bd59ebd32bb1d9511f14ff2df1b8bb62eae788"} Nov 02 23:22:58.968933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:58.968509 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:22:59.076948 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:22:59.076887 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" podStartSLOduration=2.076873181 podStartE2EDuration="2.076873181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:59.07450786 +0000 UTC m=+1772.481819691" watchObservedRunningTime="2026-11-02 23:22:59.076873181 +0000 UTC m=+1772.484185011" Nov 02 23:23:03.920757 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:03.920727 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:23:03.921300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:03.920807 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:23:07.461977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:07.461935 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv"] Nov 02 23:23:07.462957 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:07.462228 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" containerID="cri-o://45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9" gracePeriod=30 Nov 02 23:23:09.978887 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:09.978859 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:23:12.268246 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.268196 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:23:12.268831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.268281 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:23:12.738432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.738404 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:23:12.793587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.793558 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-certs\") pod \"24fb0d07-d92b-4378-a563-557e881fd75a\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " Nov 02 23:23:12.793754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.793605 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24fb0d07-d92b-4378-a563-557e881fd75a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fb0d07-d92b-4378-a563-557e881fd75a-istio-podinfo\") pod \"24fb0d07-d92b-4378-a563-557e881fd75a\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " Nov 02 23:23:12.793754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.793676 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-envoy\") pod \"24fb0d07-d92b-4378-a563-557e881fd75a\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " Nov 02 23:23:12.793754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.793703 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-socket\") pod \"24fb0d07-d92b-4378-a563-557e881fd75a\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " Nov 02 23:23:12.793890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.793755 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-kube-api-access-9h5lk\" (UniqueName: \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-kube-api-access-9h5lk\") pod \"24fb0d07-d92b-4378-a563-557e881fd75a\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " Nov 02 23:23:12.793890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.793836 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-data\") pod \"24fb0d07-d92b-4378-a563-557e881fd75a\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " Nov 02 23:23:12.793890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.793863 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-certs" pod "24fb0d07-d92b-4378-a563-557e881fd75a" (UID: "24fb0d07-d92b-4378-a563-557e881fd75a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:12.794036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.793879 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-istio-token\" (UniqueName: \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-istio-token\") pod \"24fb0d07-d92b-4378-a563-557e881fd75a\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " Nov 02 23:23:12.794036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.794004 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-credential-socket\") pod \"24fb0d07-d92b-4378-a563-557e881fd75a\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " Nov 02 23:23:12.794036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.794024 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-socket" pod "24fb0d07-d92b-4378-a563-557e881fd75a" (UID: "24fb0d07-d92b-4378-a563-557e881fd75a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:12.794222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.794111 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24fb0d07-d92b-4378-a563-557e881fd75a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fb0d07-d92b-4378-a563-557e881fd75a-istiod-ca-cert\") pod \"24fb0d07-d92b-4378-a563-557e881fd75a\" (UID: \"24fb0d07-d92b-4378-a563-557e881fd75a\") " Nov 02 23:23:12.794222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.794144 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-data" pod "24fb0d07-d92b-4378-a563-557e881fd75a" (UID: "24fb0d07-d92b-4378-a563-557e881fd75a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:12.794357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.794300 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-credential-socket" pod "24fb0d07-d92b-4378-a563-557e881fd75a" (UID: "24fb0d07-d92b-4378-a563-557e881fd75a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:12.794677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.794590 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fb0d07-d92b-4378-a563-557e881fd75a-istiod-ca-cert" pod "24fb0d07-d92b-4378-a563-557e881fd75a" (UID: "24fb0d07-d92b-4378-a563-557e881fd75a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:23:12.794677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.794640 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:12.794677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.794656 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:12.794677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.794666 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:12.794677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.794675 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:12.796080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.796053 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24fb0d07-d92b-4378-a563-557e881fd75a-istio-podinfo" pod "24fb0d07-d92b-4378-a563-557e881fd75a" (UID: "24fb0d07-d92b-4378-a563-557e881fd75a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:23:12.796201 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.796125 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-kube-api-access-9h5lk" pod "24fb0d07-d92b-4378-a563-557e881fd75a" (UID: "24fb0d07-d92b-4378-a563-557e881fd75a"). InnerVolumeSpecName "kube-api-access-9h5lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:12.796321 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.796299 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-istio-token" pod "24fb0d07-d92b-4378-a563-557e881fd75a" (UID: "24fb0d07-d92b-4378-a563-557e881fd75a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:12.796383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.796357 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-envoy" pod "24fb0d07-d92b-4378-a563-557e881fd75a" (UID: "24fb0d07-d92b-4378-a563-557e881fd75a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:12.895637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.895555 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24fb0d07-d92b-4378-a563-557e881fd75a-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:12.895637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.895584 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h5lk\" (UniqueName: \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-kube-api-access-9h5lk\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:12.895637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.895596 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24fb0d07-d92b-4378-a563-557e881fd75a-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:12.895637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.895606 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24fb0d07-d92b-4378-a563-557e881fd75a-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:12.895637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:12.895616 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24fb0d07-d92b-4378-a563-557e881fd75a-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:13.086212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:13.086179 5672 generic.go:356] "Generic (PLEG): container finished" podID="24fb0d07-d92b-4378-a563-557e881fd75a" containerID="45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9" exitCode=0 Nov 02 23:23:13.086373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:13.086237 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" Nov 02 23:23:13.086373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:13.086277 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" event={"ID":"24fb0d07-d92b-4378-a563-557e881fd75a","Type":"ContainerDied","Data":"45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9"} Nov 02 23:23:13.086373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:13.086338 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv" event={"ID":"24fb0d07-d92b-4378-a563-557e881fd75a","Type":"ContainerDied","Data":"4b08c39026c90fc12ae129b4ad7d7933409535039ee7e14845851735016d605a"} Nov 02 23:23:13.086373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:13.086361 5672 scope.go:122] "RemoveContainer" containerID="45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9" Nov 02 23:23:13.097874 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:13.097855 5672 scope.go:122] "RemoveContainer" containerID="45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9" Nov 02 23:23:13.098206 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:23:13.098180 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9\": container with ID starting with 45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9 not found: ID does not exist" containerID="45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9" Nov 02 23:23:13.098310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:13.098214 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9"} err="failed to get container status \"45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9\": rpc error: code = NotFound desc = could not find container \"45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9\": container with ID starting with 45c794443c05f6826fb1acb3d2e443a4dc99c2d203f825782076dc9dfebc94a9 not found: ID does not exist" Nov 02 23:23:13.250175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:13.248225 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv"] Nov 02 23:23:13.271172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:13.261065 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-gn5bv"] Nov 02 23:23:14.713163 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:14.713113 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" path="/var/lib/kubelet/pods/24fb0d07-d92b-4378-a563-557e881fd75a/volumes" Nov 02 23:23:17.076513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:17.076479 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:23:17.549243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:17.546735 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b"] Nov 02 23:23:17.549243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:17.546984 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" containerID="cri-o://b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb" gracePeriod=30 Nov 02 23:23:20.673336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.673291 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:23:20.673843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.673378 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:23:20.737021 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.736980 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx"] Nov 02 23:23:20.737800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.737775 5672 cpu_manager.go:414] "containerMap: removing container" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" Nov 02 23:23:20.737954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.737836 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" Nov 02 23:23:20.738052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.738037 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="24fb0d07-d92b-4378-a563-557e881fd75a" containerName="istio-proxy" Nov 02 23:23:20.741962 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.741933 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:23:20.791143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.791099 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-981724\"" Nov 02 23:23:20.814656 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.814615 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqmj\" (UniqueName: \"kubernetes.io/projected/97b25fb9-fb60-4467-ae20-958bb09a5bfa-kube-api-access-wrqmj\") pod \"payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx\" (UID: \"97b25fb9-fb60-4467-ae20-958bb09a5bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:23:20.814832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.814808 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b25fb9-fb60-4467-ae20-958bb09a5bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx\" (UID: \"97b25fb9-fb60-4467-ae20-958bb09a5bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:23:20.834503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.834459 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx"] Nov 02 23:23:20.837377 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.837349 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-981724-dockercfg-7b5g5\"" Nov 02 23:23:20.919853 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.919815 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqmj\" (UniqueName: \"kubernetes.io/projected/97b25fb9-fb60-4467-ae20-958bb09a5bfa-kube-api-access-wrqmj\") pod \"payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx\" (UID: \"97b25fb9-fb60-4467-ae20-958bb09a5bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:23:20.920491 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.920461 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b25fb9-fb60-4467-ae20-958bb09a5bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx\" (UID: \"97b25fb9-fb60-4467-ae20-958bb09a5bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:23:20.921373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.921348 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b25fb9-fb60-4467-ae20-958bb09a5bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx\" (UID: \"97b25fb9-fb60-4467-ae20-958bb09a5bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:23:20.994630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:20.994588 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqmj\" (UniqueName: \"kubernetes.io/projected/97b25fb9-fb60-4467-ae20-958bb09a5bfa-kube-api-access-wrqmj\") pod \"payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx\" (UID: \"97b25fb9-fb60-4467-ae20-958bb09a5bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:23:21.057793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:21.057759 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:23:21.428430 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:21.428406 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx"] Nov 02 23:23:21.429091 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:23:21.429061 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b25fb9_fb60_4467_ae20_958bb09a5bfa.slice/crio-e45ab3aa249ea54e4fcd84b5044c93b263a7cbcd13258935994b10fb497d0cff WatchSource:0}: Error finding container e45ab3aa249ea54e4fcd84b5044c93b263a7cbcd13258935994b10fb497d0cff: Status 404 returned error can't find the container with id e45ab3aa249ea54e4fcd84b5044c93b263a7cbcd13258935994b10fb497d0cff Nov 02 23:23:22.153475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.153422 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" event={"ID":"97b25fb9-fb60-4467-ae20-958bb09a5bfa","Type":"ContainerStarted","Data":"34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798"} Nov 02 23:23:22.154083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.153492 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" event={"ID":"97b25fb9-fb60-4467-ae20-958bb09a5bfa","Type":"ContainerStarted","Data":"e45ab3aa249ea54e4fcd84b5044c93b263a7cbcd13258935994b10fb497d0cff"} Nov 02 23:23:22.154083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.153668 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:23:22.260422 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.260359 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" podStartSLOduration=2.260341717 podStartE2EDuration="2.260341717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:22.259122527 +0000 UTC m=+1795.666434357" watchObservedRunningTime="2026-11-02 23:23:22.260341717 +0000 UTC m=+1795.667653549" Nov 02 23:23:22.838254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.838221 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:23:22.960109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.960022 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-podinfo\") pod \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " Nov 02 23:23:22.960109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.960062 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istiod-ca-cert\") pod \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " Nov 02 23:23:22.960359 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.960110 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-token\" (UniqueName: \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-token\") pod \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " Nov 02 23:23:22.960359 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.960207 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-data\") pod \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " Nov 02 23:23:22.960359 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.960266 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-envoy\") pod \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " Nov 02 23:23:22.960516 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.960357 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-credential-socket\") pod \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " Nov 02 23:23:22.960516 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.960389 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-kube-api-access-94vl6\" (UniqueName: \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-kube-api-access-94vl6\") pod \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " Nov 02 23:23:22.960516 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.960426 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-socket\") pod \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " Nov 02 23:23:22.960516 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.960488 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-certs\") pod \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\" (UID: \"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb\") " Nov 02 23:23:22.960708 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.960568 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istiod-ca-cert" pod "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" (UID: "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:23:22.961214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.960982 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:22.961214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.961019 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-socket" pod "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" (UID: "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:22.961214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.961081 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-credential-socket" pod "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" (UID: "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:22.961469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.961261 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-data" pod "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" (UID: "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:22.961622 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.961583 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-certs" pod "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" (UID: "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:22.962763 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.962734 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-podinfo" pod "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" (UID: "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:23:22.963065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.963042 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-envoy" pod "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" (UID: "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:22.963504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.963478 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-kube-api-access-94vl6" pod "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" (UID: "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb"). InnerVolumeSpecName "kube-api-access-94vl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:22.966610 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:22.966583 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-token" pod "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" (UID: "7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:23.061787 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.061744 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:23.061787 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.061781 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:23.061787 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.061794 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:23.062058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.061806 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:23.062058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.061818 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:23.062058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.061830 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:23.062058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.061842 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94vl6\" (UniqueName: \"kubernetes.io/projected/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-kube-api-access-94vl6\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:23.062058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.061855 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:23.171347 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.171310 5672 generic.go:356] "Generic (PLEG): container finished" podID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerID="b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb" exitCode=0 Nov 02 23:23:23.171812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.171429 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" Nov 02 23:23:23.171812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.171453 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" event={"ID":"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb","Type":"ContainerDied","Data":"b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb"} Nov 02 23:23:23.171812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.171522 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b" event={"ID":"7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb","Type":"ContainerDied","Data":"66d7715530a25a432eb6ca725cf62d3130e92999cadb63113fca957a00250faf"} Nov 02 23:23:23.171812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.171574 5672 scope.go:122] "RemoveContainer" containerID="b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb" Nov 02 23:23:23.182692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.182674 5672 scope.go:122] "RemoveContainer" containerID="b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb" Nov 02 23:23:23.182994 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:23:23.182976 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb\": container with ID starting with b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb not found: ID does not exist" containerID="b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb" Nov 02 23:23:23.183057 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.183006 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb"} err="failed to get container status \"b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb\": rpc error: code = NotFound desc = could not find container \"b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb\": container with ID starting with b9492f98337c3d12d0827e074718deca96762ddad671c9881e2f92195275d5bb not found: ID does not exist" Nov 02 23:23:23.325901 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.325860 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b"] Nov 02 23:23:23.343170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:23.339927 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-qjv8b"] Nov 02 23:23:24.713326 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:24.713288 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" path="/var/lib/kubelet/pods/7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb/volumes" Nov 02 23:23:26.021340 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:23:26.021293 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:23:27.864920 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:27.864887 5672 scope.go:122] "RemoveContainer" containerID="218049555efe0523d72fdf3be6fc7cb658601f5ada3a3d21111cb651ce8c172b" Nov 02 23:23:27.874387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:27.874363 5672 scope.go:122] "RemoveContainer" containerID="f5297a64f93827d18646f02cb51bacf7016b651477e559cdb7c69b52eea56bb0" Nov 02 23:23:27.884210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:27.884181 5672 scope.go:122] "RemoveContainer" containerID="024163a7b5327400c7e3963d8b9811a5f54334a811871d4af668bdd4e42184c0" Nov 02 23:23:32.129487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:32.129445 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh"] Nov 02 23:23:32.129959 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:32.129689 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" containerID="cri-o://729ad55d3e047625dcb0d3c4c005c735875807d46630f601f2f195239453a3c1" gracePeriod=30 Nov 02 23:23:32.175842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:32.175806 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h"] Nov 02 23:23:32.176045 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:32.176011 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" podUID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" containerName="istio-proxy" containerID="cri-o://564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6" gracePeriod=30 Nov 02 23:23:33.176378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:33.176348 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:23:37.265327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.265285 5672 generic.go:356] "Generic (PLEG): container finished" podID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerID="729ad55d3e047625dcb0d3c4c005c735875807d46630f601f2f195239453a3c1" exitCode=0 Nov 02 23:23:37.265750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.265357 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" event={"ID":"2d6ebfb5-2a91-4f51-aef7-205771fb098f","Type":"ContainerDied","Data":"729ad55d3e047625dcb0d3c4c005c735875807d46630f601f2f195239453a3c1"} Nov 02 23:23:37.462623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.462596 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:23:37.491081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.491051 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:23:37.544471 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544430 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-socket\") pod \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " Nov 02 23:23:37.544471 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544477 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-kube-api-access-5hfct\" (UniqueName: \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-kube-api-access-5hfct\") pod \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " Nov 02 23:23:37.544751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544520 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-kube-api-access-pplmp\" (UniqueName: \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-kube-api-access-pplmp\") pod \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " Nov 02 23:23:37.544751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544545 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-data\") pod \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " Nov 02 23:23:37.544751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544572 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-socket\") pod \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " Nov 02 23:23:37.544751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544613 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-token\" (UniqueName: \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-token\") pod \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " Nov 02 23:23:37.544751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544695 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-credential-socket\") pod \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " Nov 02 23:23:37.544751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544725 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-data\") pod \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " Nov 02 23:23:37.545062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544790 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-token\" (UniqueName: \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-token\") pod \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " Nov 02 23:23:37.545062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544813 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-socket" pod "2d6ebfb5-2a91-4f51-aef7-205771fb098f" (UID: "2d6ebfb5-2a91-4f51-aef7-205771fb098f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:37.545062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544840 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-podinfo\") pod \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " Nov 02 23:23:37.545062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544893 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istiod-ca-cert\") pod \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " Nov 02 23:23:37.545062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544922 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-credential-socket\") pod \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " Nov 02 23:23:37.545062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.544960 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-certs\") pod \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " Nov 02 23:23:37.545062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545022 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-envoy\") pod \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " Nov 02 23:23:37.545461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545051 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-data" pod "2d6ebfb5-2a91-4f51-aef7-205771fb098f" (UID: "2d6ebfb5-2a91-4f51-aef7-205771fb098f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:37.545461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545104 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-data" pod "351d4c63-e1a7-4f82-a155-1c43ebdf3311" (UID: "351d4c63-e1a7-4f82-a155-1c43ebdf3311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:37.545461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545117 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-envoy\") pod \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " Nov 02 23:23:37.545461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545166 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-certs\") pod \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " Nov 02 23:23:37.545461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545198 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-podinfo\") pod \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\" (UID: \"351d4c63-e1a7-4f82-a155-1c43ebdf3311\") " Nov 02 23:23:37.545461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545235 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istiod-ca-cert\") pod \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\" (UID: \"2d6ebfb5-2a91-4f51-aef7-205771fb098f\") " Nov 02 23:23:37.545461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545290 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-credential-socket" pod "2d6ebfb5-2a91-4f51-aef7-205771fb098f" (UID: "2d6ebfb5-2a91-4f51-aef7-205771fb098f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:37.545461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545305 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-socket" pod "351d4c63-e1a7-4f82-a155-1c43ebdf3311" (UID: "351d4c63-e1a7-4f82-a155-1c43ebdf3311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:37.545865 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545591 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istiod-ca-cert" pod "351d4c63-e1a7-4f82-a155-1c43ebdf3311" (UID: "351d4c63-e1a7-4f82-a155-1c43ebdf3311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:23:37.545865 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545766 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-credential-socket" pod "351d4c63-e1a7-4f82-a155-1c43ebdf3311" (UID: "351d4c63-e1a7-4f82-a155-1c43ebdf3311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:37.545985 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545917 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.545985 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545939 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.545985 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545954 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.545985 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545969 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.545985 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545975 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-certs" pod "2d6ebfb5-2a91-4f51-aef7-205771fb098f" (UID: "2d6ebfb5-2a91-4f51-aef7-205771fb098f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:37.545985 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.545982 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.546356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.546002 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istiod-ca-cert" pod "2d6ebfb5-2a91-4f51-aef7-205771fb098f" (UID: "2d6ebfb5-2a91-4f51-aef7-205771fb098f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:23:37.546356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.546009 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.546356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.546036 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.546515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.546470 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-certs" pod "351d4c63-e1a7-4f82-a155-1c43ebdf3311" (UID: "351d4c63-e1a7-4f82-a155-1c43ebdf3311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:37.548077 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.548047 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-token" pod "2d6ebfb5-2a91-4f51-aef7-205771fb098f" (UID: "2d6ebfb5-2a91-4f51-aef7-205771fb098f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:37.548403 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.548376 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-envoy" pod "351d4c63-e1a7-4f82-a155-1c43ebdf3311" (UID: "351d4c63-e1a7-4f82-a155-1c43ebdf3311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:37.548585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.548562 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-kube-api-access-pplmp" pod "351d4c63-e1a7-4f82-a155-1c43ebdf3311" (UID: "351d4c63-e1a7-4f82-a155-1c43ebdf3311"). InnerVolumeSpecName "kube-api-access-pplmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:37.548660 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.548616 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-token" pod "351d4c63-e1a7-4f82-a155-1c43ebdf3311" (UID: "351d4c63-e1a7-4f82-a155-1c43ebdf3311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:37.549067 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.549031 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-podinfo" pod "2d6ebfb5-2a91-4f51-aef7-205771fb098f" (UID: "2d6ebfb5-2a91-4f51-aef7-205771fb098f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:23:37.549067 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.549054 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-kube-api-access-5hfct" pod "2d6ebfb5-2a91-4f51-aef7-205771fb098f" (UID: "2d6ebfb5-2a91-4f51-aef7-205771fb098f"). InnerVolumeSpecName "kube-api-access-5hfct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:23:37.549286 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.549268 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-envoy" pod "2d6ebfb5-2a91-4f51-aef7-205771fb098f" (UID: "2d6ebfb5-2a91-4f51-aef7-205771fb098f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:23:37.549941 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.549921 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-podinfo" pod "351d4c63-e1a7-4f82-a155-1c43ebdf3311" (UID: "351d4c63-e1a7-4f82-a155-1c43ebdf3311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:23:37.648048 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.648010 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.648375 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.648319 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.648519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.648495 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.648519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.648521 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.648682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.648536 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.648682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.648551 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.648682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.648568 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351d4c63-e1a7-4f82-a155-1c43ebdf3311-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.648682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.648581 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351d4c63-e1a7-4f82-a155-1c43ebdf3311-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.648682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.648595 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d6ebfb5-2a91-4f51-aef7-205771fb098f-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.648682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.648611 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hfct\" (UniqueName: \"kubernetes.io/projected/2d6ebfb5-2a91-4f51-aef7-205771fb098f-kube-api-access-5hfct\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:37.648682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:37.648626 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pplmp\" (UniqueName: \"kubernetes.io/projected/351d4c63-e1a7-4f82-a155-1c43ebdf3311-kube-api-access-pplmp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:23:38.275756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.275723 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" Nov 02 23:23:38.275756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.275735 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh" event={"ID":"2d6ebfb5-2a91-4f51-aef7-205771fb098f","Type":"ContainerDied","Data":"3221c95cbf37cfab78095cd1fe50abf6c18397f4175389961098f29dd36906c4"} Nov 02 23:23:38.276300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.275802 5672 scope.go:122] "RemoveContainer" containerID="729ad55d3e047625dcb0d3c4c005c735875807d46630f601f2f195239453a3c1" Nov 02 23:23:38.277494 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.277465 5672 generic.go:356] "Generic (PLEG): container finished" podID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" containerID="564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6" exitCode=0 Nov 02 23:23:38.277604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.277524 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" Nov 02 23:23:38.277604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.277542 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" event={"ID":"351d4c63-e1a7-4f82-a155-1c43ebdf3311","Type":"ContainerDied","Data":"564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6"} Nov 02 23:23:38.277604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.277592 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h" event={"ID":"351d4c63-e1a7-4f82-a155-1c43ebdf3311","Type":"ContainerDied","Data":"80b2ad27f9a47cca3c056c915166696de30567853133bb84b90bddd4b65916d1"} Nov 02 23:23:38.287223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.287063 5672 scope.go:122] "RemoveContainer" containerID="564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6" Nov 02 23:23:38.297108 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.297084 5672 scope.go:122] "RemoveContainer" containerID="564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6" Nov 02 23:23:38.297441 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:23:38.297419 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6\": container with ID starting with 564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6 not found: ID does not exist" containerID="564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6" Nov 02 23:23:38.297486 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.297450 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6"} err="failed to get container status \"564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6\": rpc error: code = NotFound desc = could not find container \"564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6\": container with ID starting with 564f5610ea65d0e47e00898f47ac9f8edb5675c4cc21557b3f3275b1aaae09d6 not found: ID does not exist" Nov 02 23:23:38.480029 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.479993 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh"] Nov 02 23:23:38.525294 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.525260 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-vq9vh"] Nov 02 23:23:38.585815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.585768 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h"] Nov 02 23:23:38.589907 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.589880 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-b5q9h"] Nov 02 23:23:38.713508 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.713464 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" path="/var/lib/kubelet/pods/2d6ebfb5-2a91-4f51-aef7-205771fb098f/volumes" Nov 02 23:23:38.713944 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:38.713931 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" path="/var/lib/kubelet/pods/351d4c63-e1a7-4f82-a155-1c43ebdf3311/volumes" Nov 02 23:23:41.046296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:41.046249 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9"] Nov 02 23:23:41.091295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:41.091258 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff0eef6-mcbx9"] Nov 02 23:23:42.147824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:42.147784 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:23:42.713311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:23:42.713273 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdfd4a95-9541-430b-baee-364f8c1b5da1" path="/var/lib/kubelet/pods/cdfd4a95-9541-430b-baee-364f8c1b5da1/volumes" Nov 02 23:23:56.021118 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:23:56.021075 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:24:18.715585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:18.715551 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg"] Nov 02 23:24:18.716012 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:18.715759 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" containerID="cri-o://b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf" gracePeriod=30 Nov 02 23:24:18.761558 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:18.761523 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts"] Nov 02 23:24:18.761738 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:18.761702 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" containerID="cri-o://b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de" gracePeriod=30 Nov 02 23:24:23.078707 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:23.078664 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:24:23.079121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:23.078740 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:24:23.272003 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:23.271963 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:24:23.272215 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:23.272019 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:24:24.063020 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.062985 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:24:24.102809 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.102778 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:24:24.219015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.218975 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-podinfo\") pod \"5a976591-e56a-4344-afbf-1a0c369a10d1\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " Nov 02 23:24:24.219244 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219056 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-certs\") pod \"5a976591-e56a-4344-afbf-1a0c369a10d1\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " Nov 02 23:24:24.219244 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219096 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-kube-api-access-6zdf9\" (UniqueName: \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-kube-api-access-6zdf9\") pod \"5a976591-e56a-4344-afbf-1a0c369a10d1\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " Nov 02 23:24:24.219244 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219127 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-token\" (UniqueName: \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-token\") pod \"5a976591-e56a-4344-afbf-1a0c369a10d1\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " Nov 02 23:24:24.219244 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219174 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-certs\") pod \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " Nov 02 23:24:24.219244 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219202 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-socket\") pod \"5a976591-e56a-4344-afbf-1a0c369a10d1\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " Nov 02 23:24:24.219244 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219230 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a976591-e56a-4344-afbf-1a0c369a10d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a976591-e56a-4344-afbf-1a0c369a10d1-istiod-ca-cert\") pod \"5a976591-e56a-4344-afbf-1a0c369a10d1\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " Nov 02 23:24:24.219590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219300 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-kube-api-access-9jn8t\" (UniqueName: \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-kube-api-access-9jn8t\") pod \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " Nov 02 23:24:24.219590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219335 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-envoy\") pod \"5a976591-e56a-4344-afbf-1a0c369a10d1\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " Nov 02 23:24:24.219590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219362 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istiod-ca-cert\") pod \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " Nov 02 23:24:24.219590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219422 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-credential-socket\") pod \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " Nov 02 23:24:24.219590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219451 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-podinfo\") pod \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " Nov 02 23:24:24.219590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219496 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-socket\") pod \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " Nov 02 23:24:24.219590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219540 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-envoy\") pod \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " Nov 02 23:24:24.219937 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219616 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-token\" (UniqueName: \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-token\") pod \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " Nov 02 23:24:24.219937 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219660 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-credential-socket\") pod \"5a976591-e56a-4344-afbf-1a0c369a10d1\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " Nov 02 23:24:24.219937 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219720 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-data\") pod \"5a976591-e56a-4344-afbf-1a0c369a10d1\" (UID: \"5a976591-e56a-4344-afbf-1a0c369a10d1\") " Nov 02 23:24:24.219937 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219751 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-data\") pod \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\" (UID: \"7fdcf2cb-a12c-4a89-b1e2-9774768069e7\") " Nov 02 23:24:24.219937 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219829 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a976591-e56a-4344-afbf-1a0c369a10d1-istiod-ca-cert" pod "5a976591-e56a-4344-afbf-1a0c369a10d1" (UID: "5a976591-e56a-4344-afbf-1a0c369a10d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:24.220229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.219933 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istiod-ca-cert" pod "7fdcf2cb-a12c-4a89-b1e2-9774768069e7" (UID: "7fdcf2cb-a12c-4a89-b1e2-9774768069e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:24.220229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.220069 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-certs" pod "7fdcf2cb-a12c-4a89-b1e2-9774768069e7" (UID: "7fdcf2cb-a12c-4a89-b1e2-9774768069e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.220229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.220192 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-data" pod "7fdcf2cb-a12c-4a89-b1e2-9774768069e7" (UID: "7fdcf2cb-a12c-4a89-b1e2-9774768069e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.220380 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.220349 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-socket" pod "5a976591-e56a-4344-afbf-1a0c369a10d1" (UID: "5a976591-e56a-4344-afbf-1a0c369a10d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.220492 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.220467 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.220546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.220494 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.220546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.220511 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a976591-e56a-4344-afbf-1a0c369a10d1-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.220546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.220526 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.220546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.220542 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.220720 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.220556 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-credential-socket" pod "5a976591-e56a-4344-afbf-1a0c369a10d1" (UID: "5a976591-e56a-4344-afbf-1a0c369a10d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.220775 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.220733 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-data" pod "5a976591-e56a-4344-afbf-1a0c369a10d1" (UID: "5a976591-e56a-4344-afbf-1a0c369a10d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.221276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.221225 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-certs" pod "5a976591-e56a-4344-afbf-1a0c369a10d1" (UID: "5a976591-e56a-4344-afbf-1a0c369a10d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.221442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.221419 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-credential-socket" pod "7fdcf2cb-a12c-4a89-b1e2-9774768069e7" (UID: "7fdcf2cb-a12c-4a89-b1e2-9774768069e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.221635 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.221611 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-socket" pod "7fdcf2cb-a12c-4a89-b1e2-9774768069e7" (UID: "7fdcf2cb-a12c-4a89-b1e2-9774768069e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.223832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.223797 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-podinfo" pod "5a976591-e56a-4344-afbf-1a0c369a10d1" (UID: "5a976591-e56a-4344-afbf-1a0c369a10d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:24:24.224082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.223620 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-kube-api-access-6zdf9" pod "5a976591-e56a-4344-afbf-1a0c369a10d1" (UID: "5a976591-e56a-4344-afbf-1a0c369a10d1"). InnerVolumeSpecName "kube-api-access-6zdf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:24.224082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.224048 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-envoy" pod "7fdcf2cb-a12c-4a89-b1e2-9774768069e7" (UID: "7fdcf2cb-a12c-4a89-b1e2-9774768069e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.224274 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.224171 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-podinfo" pod "7fdcf2cb-a12c-4a89-b1e2-9774768069e7" (UID: "7fdcf2cb-a12c-4a89-b1e2-9774768069e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:24:24.224597 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.224572 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-token" pod "7fdcf2cb-a12c-4a89-b1e2-9774768069e7" (UID: "7fdcf2cb-a12c-4a89-b1e2-9774768069e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:24.224682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.224666 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-envoy" pod "5a976591-e56a-4344-afbf-1a0c369a10d1" (UID: "5a976591-e56a-4344-afbf-1a0c369a10d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:24:24.225316 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.225281 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-kube-api-access-9jn8t" pod "7fdcf2cb-a12c-4a89-b1e2-9774768069e7" (UID: "7fdcf2cb-a12c-4a89-b1e2-9774768069e7"). InnerVolumeSpecName "kube-api-access-9jn8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:24.225702 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.225671 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-token" pod "5a976591-e56a-4344-afbf-1a0c369a10d1" (UID: "5a976591-e56a-4344-afbf-1a0c369a10d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:24.322080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.321997 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322023 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322033 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322043 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322051 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322060 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322070 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zdf9\" (UniqueName: \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-kube-api-access-6zdf9\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322079 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322088 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jn8t\" (UniqueName: \"kubernetes.io/projected/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-kube-api-access-9jn8t\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322096 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a976591-e56a-4344-afbf-1a0c369a10d1-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322105 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322113 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.322478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.322121 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdcf2cb-a12c-4a89-b1e2-9774768069e7-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:24.588075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.587989 5672 generic.go:356] "Generic (PLEG): container finished" podID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerID="b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf" exitCode=0 Nov 02 23:24:24.588075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.588063 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" Nov 02 23:24:24.588372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.588075 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" event={"ID":"7fdcf2cb-a12c-4a89-b1e2-9774768069e7","Type":"ContainerDied","Data":"b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf"} Nov 02 23:24:24.588372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.588189 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg" event={"ID":"7fdcf2cb-a12c-4a89-b1e2-9774768069e7","Type":"ContainerDied","Data":"b617184bc7e1ea7251b1b2965ecab61b09ad8bd6823b8208aeb2d0003c9ada0c"} Nov 02 23:24:24.588372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.588221 5672 scope.go:122] "RemoveContainer" containerID="b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf" Nov 02 23:24:24.589652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.589629 5672 generic.go:356] "Generic (PLEG): container finished" podID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerID="b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de" exitCode=0 Nov 02 23:24:24.589792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.589690 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" Nov 02 23:24:24.589792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.589715 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" event={"ID":"5a976591-e56a-4344-afbf-1a0c369a10d1","Type":"ContainerDied","Data":"b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de"} Nov 02 23:24:24.589792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.589767 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts" event={"ID":"5a976591-e56a-4344-afbf-1a0c369a10d1","Type":"ContainerDied","Data":"dc49dfebead99ab8399da17def0018caf6ba63cfe3a25a5ff7eb8c2ecaddcbdd"} Nov 02 23:24:24.598364 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.598348 5672 scope.go:122] "RemoveContainer" containerID="b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf" Nov 02 23:24:24.598659 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:24:24.598637 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf\": container with ID starting with b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf not found: ID does not exist" containerID="b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf" Nov 02 23:24:24.598741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.598672 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf"} err="failed to get container status \"b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf\": rpc error: code = NotFound desc = could not find container \"b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf\": container with ID starting with b5fc8d60a0f348dcccea5b87b2af60c885a16a4e269f002785e0964dd41830bf not found: ID does not exist" Nov 02 23:24:24.598741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.598697 5672 scope.go:122] "RemoveContainer" containerID="b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de" Nov 02 23:24:24.607526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.607508 5672 scope.go:122] "RemoveContainer" containerID="b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de" Nov 02 23:24:24.607754 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:24:24.607737 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de\": container with ID starting with b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de not found: ID does not exist" containerID="b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de" Nov 02 23:24:24.607812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.607762 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de"} err="failed to get container status \"b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de\": rpc error: code = NotFound desc = could not find container \"b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de\": container with ID starting with b7518735d0c508197c69898190f70bcd0f522e779a7829a03a276b64ec7fd0de not found: ID does not exist" Nov 02 23:24:24.750086 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.750051 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts"] Nov 02 23:24:24.794593 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.794562 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-phtts"] Nov 02 23:24:24.851530 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.851433 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg"] Nov 02 23:24:24.857881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:24.857852 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-rknpg"] Nov 02 23:24:25.380716 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:25.380685 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9"] Nov 02 23:24:25.482500 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:25.482468 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7"] Nov 02 23:24:25.605207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:25.605172 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:24:26.032027 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:24:26.031987 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:24:26.714206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:26.714173 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" path="/var/lib/kubelet/pods/5a976591-e56a-4344-afbf-1a0c369a10d1/volumes" Nov 02 23:24:26.714835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:26.714813 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" path="/var/lib/kubelet/pods/7fdcf2cb-a12c-4a89-b1e2-9774768069e7/volumes" Nov 02 23:24:30.381232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.381182 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" podUID="47bcaf02-f310-4d31-b36d-1765952304e7" containerName="payload-processing" containerID="cri-o://0a2f0bd020ea4d83356f3381f019c30034f1f5a85c129c607d97d0e9bcb4478d" gracePeriod=25 Nov 02 23:24:30.483443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.483399 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" podUID="96e48b30-2e0c-4996-8db6-613c2a068866" containerName="payload-pre-processing" containerID="cri-o://ee9ffb55d940aa0a2d4ff40a1620fdf0a3d0619d9375c192518eb825632872df" gracePeriod=25 Nov 02 23:24:30.634704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.631237 5672 generic.go:356] "Generic (PLEG): container finished" podID="47bcaf02-f310-4d31-b36d-1765952304e7" containerID="0a2f0bd020ea4d83356f3381f019c30034f1f5a85c129c607d97d0e9bcb4478d" exitCode=0 Nov 02 23:24:30.634704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.631436 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" event={"ID":"47bcaf02-f310-4d31-b36d-1765952304e7","Type":"ContainerDied","Data":"0a2f0bd020ea4d83356f3381f019c30034f1f5a85c129c607d97d0e9bcb4478d"} Nov 02 23:24:30.634704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.634142 5672 generic.go:356] "Generic (PLEG): container finished" podID="96e48b30-2e0c-4996-8db6-613c2a068866" containerID="ee9ffb55d940aa0a2d4ff40a1620fdf0a3d0619d9375c192518eb825632872df" exitCode=0 Nov 02 23:24:30.634704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.634312 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" event={"ID":"96e48b30-2e0c-4996-8db6-613c2a068866","Type":"ContainerDied","Data":"ee9ffb55d940aa0a2d4ff40a1620fdf0a3d0619d9375c192518eb825632872df"} Nov 02 23:24:30.673959 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.673932 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:24:30.707419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.707389 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47bcaf02-f310-4d31-b36d-1765952304e7-kube-api-access-zb45l\" (UniqueName: \"kubernetes.io/projected/47bcaf02-f310-4d31-b36d-1765952304e7-kube-api-access-zb45l\") pod \"47bcaf02-f310-4d31-b36d-1765952304e7\" (UID: \"47bcaf02-f310-4d31-b36d-1765952304e7\") " Nov 02 23:24:30.707587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.707536 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47bcaf02-f310-4d31-b36d-1765952304e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bcaf02-f310-4d31-b36d-1765952304e7-plugins-config-volume\") pod \"47bcaf02-f310-4d31-b36d-1765952304e7\" (UID: \"47bcaf02-f310-4d31-b36d-1765952304e7\") " Nov 02 23:24:30.707999 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.707974 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bcaf02-f310-4d31-b36d-1765952304e7-plugins-config-volume" pod "47bcaf02-f310-4d31-b36d-1765952304e7" (UID: "47bcaf02-f310-4d31-b36d-1765952304e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:30.709508 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.709482 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bcaf02-f310-4d31-b36d-1765952304e7-kube-api-access-zb45l" pod "47bcaf02-f310-4d31-b36d-1765952304e7" (UID: "47bcaf02-f310-4d31-b36d-1765952304e7"). InnerVolumeSpecName "kube-api-access-zb45l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:30.754369 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.754340 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:24:30.809126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.809095 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e48b30-2e0c-4996-8db6-613c2a068866-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e48b30-2e0c-4996-8db6-613c2a068866-plugins-config-volume\") pod \"96e48b30-2e0c-4996-8db6-613c2a068866\" (UID: \"96e48b30-2e0c-4996-8db6-613c2a068866\") " Nov 02 23:24:30.809311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.809208 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e48b30-2e0c-4996-8db6-613c2a068866-kube-api-access-l6qts\" (UniqueName: \"kubernetes.io/projected/96e48b30-2e0c-4996-8db6-613c2a068866-kube-api-access-l6qts\") pod \"96e48b30-2e0c-4996-8db6-613c2a068866\" (UID: \"96e48b30-2e0c-4996-8db6-613c2a068866\") " Nov 02 23:24:30.809492 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.809466 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e48b30-2e0c-4996-8db6-613c2a068866-plugins-config-volume" pod "96e48b30-2e0c-4996-8db6-613c2a068866" (UID: "96e48b30-2e0c-4996-8db6-613c2a068866"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:24:30.809567 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.809553 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e48b30-2e0c-4996-8db6-613c2a068866-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:30.809604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.809575 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bcaf02-f310-4d31-b36d-1765952304e7-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:30.809604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.809586 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb45l\" (UniqueName: \"kubernetes.io/projected/47bcaf02-f310-4d31-b36d-1765952304e7-kube-api-access-zb45l\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:30.811336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.811315 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e48b30-2e0c-4996-8db6-613c2a068866-kube-api-access-l6qts" pod "96e48b30-2e0c-4996-8db6-613c2a068866" (UID: "96e48b30-2e0c-4996-8db6-613c2a068866"). InnerVolumeSpecName "kube-api-access-l6qts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:24:30.911179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:30.911098 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6qts\" (UniqueName: \"kubernetes.io/projected/96e48b30-2e0c-4996-8db6-613c2a068866-kube-api-access-l6qts\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:24:31.642462 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:31.642429 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" Nov 02 23:24:31.642462 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:31.642438 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9" event={"ID":"47bcaf02-f310-4d31-b36d-1765952304e7","Type":"ContainerDied","Data":"387840944bf558a71f7162370da20d06e1092da1e6f4d37fb459f1078e2241e0"} Nov 02 23:24:31.643050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:31.642525 5672 scope.go:122] "RemoveContainer" containerID="0a2f0bd020ea4d83356f3381f019c30034f1f5a85c129c607d97d0e9bcb4478d" Nov 02 23:24:31.644326 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:31.644171 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" event={"ID":"96e48b30-2e0c-4996-8db6-613c2a068866","Type":"ContainerDied","Data":"e6ff181f5b3e2849643c402538e15162e7a3b7e5d1f4f4048ba6ab594231e4a2"} Nov 02 23:24:31.644326 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:31.644218 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7" Nov 02 23:24:31.653818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:31.653797 5672 scope.go:122] "RemoveContainer" containerID="ee9ffb55d940aa0a2d4ff40a1620fdf0a3d0619d9375c192518eb825632872df" Nov 02 23:24:31.794898 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:31.794862 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9"] Nov 02 23:24:31.841054 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:31.841020 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0064f-86f8df4f65-jc2d9"] Nov 02 23:24:31.942327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:31.942228 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7"] Nov 02 23:24:31.946194 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:31.946168 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0064f-796596bd5f-9njc7"] Nov 02 23:24:32.712437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:32.712406 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47bcaf02-f310-4d31-b36d-1765952304e7" path="/var/lib/kubelet/pods/47bcaf02-f310-4d31-b36d-1765952304e7/volumes" Nov 02 23:24:32.712843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:32.712828 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e48b30-2e0c-4996-8db6-613c2a068866" path="/var/lib/kubelet/pods/96e48b30-2e0c-4996-8db6-613c2a068866/volumes" Nov 02 23:24:56.021111 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:24:56.021067 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:24:57.239504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:24:57.239468 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:25:07.694645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.694552 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95"] Nov 02 23:25:07.695095 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695078 5672 cpu_manager.go:414] "containerMap: removing container" podUID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" containerName="istio-proxy" Nov 02 23:25:07.695144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695097 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" containerName="istio-proxy" Nov 02 23:25:07.695144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695109 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" Nov 02 23:25:07.695144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695114 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" Nov 02 23:25:07.695144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695123 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" Nov 02 23:25:07.695144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695128 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" Nov 02 23:25:07.695144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695136 5672 cpu_manager.go:414] "containerMap: removing container" podUID="96e48b30-2e0c-4996-8db6-613c2a068866" containerName="payload-pre-processing" Nov 02 23:25:07.695144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695142 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e48b30-2e0c-4996-8db6-613c2a068866" containerName="payload-pre-processing" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695165 5672 cpu_manager.go:414] "containerMap: removing container" podUID="47bcaf02-f310-4d31-b36d-1765952304e7" containerName="payload-processing" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695173 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47bcaf02-f310-4d31-b36d-1765952304e7" containerName="payload-processing" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695192 5672 cpu_manager.go:414] "containerMap: removing container" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695197 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695209 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695214 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695304 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a976591-e56a-4344-afbf-1a0c369a10d1" containerName="istio-proxy" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695313 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6ebfb5-2a91-4f51-aef7-205771fb098f" containerName="istio-proxy" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695327 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="351d4c63-e1a7-4f82-a155-1c43ebdf3311" containerName="istio-proxy" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695332 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7211e4c9-9a0a-4cb4-ac8d-9ab2ee2643bb" containerName="istio-proxy" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695338 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdcf2cb-a12c-4a89-b1e2-9774768069e7" containerName="istio-proxy" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695344 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e48b30-2e0c-4996-8db6-613c2a068866" containerName="payload-pre-processing" Nov 02 23:25:07.695385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.695348 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="47bcaf02-f310-4d31-b36d-1765952304e7" containerName="payload-processing" Nov 02 23:25:07.698854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.698834 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:25:07.807344 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.807305 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-443fdf\"" Nov 02 23:25:07.807344 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.807311 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-443fdf-dockercfg-hmns9\"" Nov 02 23:25:07.811376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.811350 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95"] Nov 02 23:25:07.846370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.846323 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abaf2bda-f79e-4510-9d60-a65cb7d709e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95\" (UID: \"abaf2bda-f79e-4510-9d60-a65cb7d709e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:25:07.846508 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.846444 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffflq\" (UniqueName: \"kubernetes.io/projected/abaf2bda-f79e-4510-9d60-a65cb7d709e1-kube-api-access-ffflq\") pod \"payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95\" (UID: \"abaf2bda-f79e-4510-9d60-a65cb7d709e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:25:07.855954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.855926 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp"] Nov 02 23:25:07.860262 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.860241 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:25:07.894261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.894231 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp"] Nov 02 23:25:07.947340 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.947271 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbjfr\" (UniqueName: \"kubernetes.io/projected/c05b59e1-7bb2-493e-b598-edd970ce8c2c-kube-api-access-hbjfr\") pod \"payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp\" (UID: \"c05b59e1-7bb2-493e-b598-edd970ce8c2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:25:07.947476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.947381 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05b59e1-7bb2-493e-b598-edd970ce8c2c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp\" (UID: \"c05b59e1-7bb2-493e-b598-edd970ce8c2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:25:07.947476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.947463 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abaf2bda-f79e-4510-9d60-a65cb7d709e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95\" (UID: \"abaf2bda-f79e-4510-9d60-a65cb7d709e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:25:07.947563 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.947521 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffflq\" (UniqueName: \"kubernetes.io/projected/abaf2bda-f79e-4510-9d60-a65cb7d709e1-kube-api-access-ffflq\") pod \"payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95\" (UID: \"abaf2bda-f79e-4510-9d60-a65cb7d709e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:25:07.948126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:07.948108 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abaf2bda-f79e-4510-9d60-a65cb7d709e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95\" (UID: \"abaf2bda-f79e-4510-9d60-a65cb7d709e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:25:08.000460 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.000436 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffflq\" (UniqueName: \"kubernetes.io/projected/abaf2bda-f79e-4510-9d60-a65cb7d709e1-kube-api-access-ffflq\") pod \"payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95\" (UID: \"abaf2bda-f79e-4510-9d60-a65cb7d709e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:25:08.012280 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.012256 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:25:08.048632 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.048597 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbjfr\" (UniqueName: \"kubernetes.io/projected/c05b59e1-7bb2-493e-b598-edd970ce8c2c-kube-api-access-hbjfr\") pod \"payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp\" (UID: \"c05b59e1-7bb2-493e-b598-edd970ce8c2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:25:08.048775 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.048729 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05b59e1-7bb2-493e-b598-edd970ce8c2c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp\" (UID: \"c05b59e1-7bb2-493e-b598-edd970ce8c2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:25:08.049425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.049404 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05b59e1-7bb2-493e-b598-edd970ce8c2c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp\" (UID: \"c05b59e1-7bb2-493e-b598-edd970ce8c2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:25:08.101883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.101849 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbjfr\" (UniqueName: \"kubernetes.io/projected/c05b59e1-7bb2-493e-b598-edd970ce8c2c-kube-api-access-hbjfr\") pod \"payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp\" (UID: \"c05b59e1-7bb2-493e-b598-edd970ce8c2c\") " pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:25:08.173646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.173612 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:25:08.382557 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.382532 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95"] Nov 02 23:25:08.384489 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:25:08.384451 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabaf2bda_f79e_4510_9d60_a65cb7d709e1.slice/crio-e0fb572c115453ba84c2aa1b8e6e79da7aad079357c114902f6c5b8952666bc2 WatchSource:0}: Error finding container e0fb572c115453ba84c2aa1b8e6e79da7aad079357c114902f6c5b8952666bc2: Status 404 returned error can't find the container with id e0fb572c115453ba84c2aa1b8e6e79da7aad079357c114902f6c5b8952666bc2 Nov 02 23:25:08.538323 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.538297 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp"] Nov 02 23:25:08.540073 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:25:08.540044 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05b59e1_7bb2_493e_b598_edd970ce8c2c.slice/crio-bb3cd6b79e2c6fe82d15c7710cfffcdd5530b10b62f92d756e4f384fee6dbc51 WatchSource:0}: Error finding container bb3cd6b79e2c6fe82d15c7710cfffcdd5530b10b62f92d756e4f384fee6dbc51: Status 404 returned error can't find the container with id bb3cd6b79e2c6fe82d15c7710cfffcdd5530b10b62f92d756e4f384fee6dbc51 Nov 02 23:25:08.894252 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.894133 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" event={"ID":"c05b59e1-7bb2-493e-b598-edd970ce8c2c","Type":"ContainerStarted","Data":"f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7"} Nov 02 23:25:08.894252 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.894242 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" event={"ID":"c05b59e1-7bb2-493e-b598-edd970ce8c2c","Type":"ContainerStarted","Data":"bb3cd6b79e2c6fe82d15c7710cfffcdd5530b10b62f92d756e4f384fee6dbc51"} Nov 02 23:25:08.894748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.894344 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:25:08.895799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.895769 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" event={"ID":"abaf2bda-f79e-4510-9d60-a65cb7d709e1","Type":"ContainerStarted","Data":"1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab"} Nov 02 23:25:08.895933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.895824 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" event={"ID":"abaf2bda-f79e-4510-9d60-a65cb7d709e1","Type":"ContainerStarted","Data":"e0fb572c115453ba84c2aa1b8e6e79da7aad079357c114902f6c5b8952666bc2"} Nov 02 23:25:08.895933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:08.895855 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:25:09.022768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:09.022719 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" podStartSLOduration=2.022706421 podStartE2EDuration="2.022706421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:09.020936552 +0000 UTC m=+1902.428248382" watchObservedRunningTime="2026-11-02 23:25:09.022706421 +0000 UTC m=+1902.430018273" Nov 02 23:25:09.132932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:09.132879 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" podStartSLOduration=2.132866292 podStartE2EDuration="2.132866292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:09.131933413 +0000 UTC m=+1902.539245242" watchObservedRunningTime="2026-11-02 23:25:09.132866292 +0000 UTC m=+1902.540178121" Nov 02 23:25:12.936896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:12.936861 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql"] Nov 02 23:25:17.937888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:17.937837 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" podUID="f80a5bc1-7bcc-4839-84c7-28253d42a95d" containerName="payload-pre-processing" containerID="cri-o://c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35" gracePeriod=25 Nov 02 23:25:18.203468 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.203407 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:25:18.372955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.372922 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80a5bc1-7bcc-4839-84c7-28253d42a95d-kube-api-access-wb948\" (UniqueName: \"kubernetes.io/projected/f80a5bc1-7bcc-4839-84c7-28253d42a95d-kube-api-access-wb948\") pod \"f80a5bc1-7bcc-4839-84c7-28253d42a95d\" (UID: \"f80a5bc1-7bcc-4839-84c7-28253d42a95d\") " Nov 02 23:25:18.373161 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.372988 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f80a5bc1-7bcc-4839-84c7-28253d42a95d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80a5bc1-7bcc-4839-84c7-28253d42a95d-plugins-config-volume\") pod \"f80a5bc1-7bcc-4839-84c7-28253d42a95d\" (UID: \"f80a5bc1-7bcc-4839-84c7-28253d42a95d\") " Nov 02 23:25:18.373473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.373447 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80a5bc1-7bcc-4839-84c7-28253d42a95d-plugins-config-volume" pod "f80a5bc1-7bcc-4839-84c7-28253d42a95d" (UID: "f80a5bc1-7bcc-4839-84c7-28253d42a95d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:18.375080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.375056 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80a5bc1-7bcc-4839-84c7-28253d42a95d-kube-api-access-wb948" pod "f80a5bc1-7bcc-4839-84c7-28253d42a95d" (UID: "f80a5bc1-7bcc-4839-84c7-28253d42a95d"). InnerVolumeSpecName "kube-api-access-wb948". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:18.474423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.474396 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb948\" (UniqueName: \"kubernetes.io/projected/f80a5bc1-7bcc-4839-84c7-28253d42a95d-kube-api-access-wb948\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:18.474423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.474423 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80a5bc1-7bcc-4839-84c7-28253d42a95d-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:18.967529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.967438 5672 generic.go:356] "Generic (PLEG): container finished" podID="f80a5bc1-7bcc-4839-84c7-28253d42a95d" containerID="c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35" exitCode=0 Nov 02 23:25:18.967529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.967491 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" Nov 02 23:25:18.967529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.967516 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" event={"ID":"f80a5bc1-7bcc-4839-84c7-28253d42a95d","Type":"ContainerDied","Data":"c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35"} Nov 02 23:25:18.968170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.967602 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql" event={"ID":"f80a5bc1-7bcc-4839-84c7-28253d42a95d","Type":"ContainerDied","Data":"a1aa49622953a20e19d79ea848bd59ebd32bb1d9511f14ff2df1b8bb62eae788"} Nov 02 23:25:18.968170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.967624 5672 scope.go:122] "RemoveContainer" containerID="c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35" Nov 02 23:25:18.981117 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.981025 5672 scope.go:122] "RemoveContainer" containerID="c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35" Nov 02 23:25:18.981544 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:25:18.981510 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35\": container with ID starting with c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35 not found: ID does not exist" containerID="c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35" Nov 02 23:25:18.981643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:18.981553 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35"} err="failed to get container status \"c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35\": rpc error: code = NotFound desc = could not find container \"c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35\": container with ID starting with c477f9af25a31e55d15bd2996daac64a46973ce4e4c72d90b54eb9bccf625c35 not found: ID does not exist" Nov 02 23:25:19.120638 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:19.120601 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql"] Nov 02 23:25:19.166707 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:19.166675 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f5790-765555656c-2xlql"] Nov 02 23:25:19.905391 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:19.905363 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:25:19.905567 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:19.905435 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:25:20.712824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:20.712792 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80a5bc1-7bcc-4839-84c7-28253d42a95d" path="/var/lib/kubelet/pods/f80a5bc1-7bcc-4839-84c7-28253d42a95d/volumes" Nov 02 23:25:22.115609 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.115578 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld"] Nov 02 23:25:22.116075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.116062 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f80a5bc1-7bcc-4839-84c7-28253d42a95d" containerName="payload-pre-processing" Nov 02 23:25:22.116124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.116076 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80a5bc1-7bcc-4839-84c7-28253d42a95d" containerName="payload-pre-processing" Nov 02 23:25:22.116172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.116165 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80a5bc1-7bcc-4839-84c7-28253d42a95d" containerName="payload-pre-processing" Nov 02 23:25:22.120584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.120565 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:25:22.219651 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.219610 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld\" (UID: \"58fff7c5-ccef-4ee3-a661-300ef85fbfe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:25:22.224595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.224565 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w27m6\" (UniqueName: \"kubernetes.io/projected/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-kube-api-access-w27m6\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld\" (UID: \"58fff7c5-ccef-4ee3-a661-300ef85fbfe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:25:22.225106 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.223326 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7fdfca-dockercfg-v56dz\"" Nov 02 23:25:22.225267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.220025 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7fdfca\"" Nov 02 23:25:22.231089 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.231059 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld"] Nov 02 23:25:22.278242 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.278212 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6"] Nov 02 23:25:22.282156 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.282125 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:25:22.326579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.326548 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6\" (UID: \"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:25:22.326723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.326598 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld\" (UID: \"58fff7c5-ccef-4ee3-a661-300ef85fbfe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:25:22.326723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.326675 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w27m6\" (UniqueName: \"kubernetes.io/projected/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-kube-api-access-w27m6\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld\" (UID: \"58fff7c5-ccef-4ee3-a661-300ef85fbfe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:25:22.326862 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.326737 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vxt\" (UniqueName: \"kubernetes.io/projected/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-kube-api-access-m4vxt\") pod \"payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6\" (UID: \"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:25:22.327428 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.327403 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld\" (UID: \"58fff7c5-ccef-4ee3-a661-300ef85fbfe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:25:22.333835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.333810 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6"] Nov 02 23:25:22.379797 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.379733 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w27m6\" (UniqueName: \"kubernetes.io/projected/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-kube-api-access-w27m6\") pod \"payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld\" (UID: \"58fff7c5-ccef-4ee3-a661-300ef85fbfe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:25:22.427615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.427583 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6\" (UID: \"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:25:22.427747 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.427632 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vxt\" (UniqueName: \"kubernetes.io/projected/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-kube-api-access-m4vxt\") pod \"payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6\" (UID: \"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:25:22.428307 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.428289 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6\" (UID: \"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:25:22.434039 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.434019 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:25:22.481287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.481264 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vxt\" (UniqueName: \"kubernetes.io/projected/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-kube-api-access-m4vxt\") pod \"payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6\" (UID: \"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:25:22.595600 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.595570 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:25:22.802703 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.802679 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld"] Nov 02 23:25:22.804466 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:25:22.804433 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fff7c5_ccef_4ee3_a661_300ef85fbfe6.slice/crio-cf6193450227f9f4f242042ed4f7f2b320ab85f497b85dd7143e7ce64ecf980e WatchSource:0}: Error finding container cf6193450227f9f4f242042ed4f7f2b320ab85f497b85dd7143e7ce64ecf980e: Status 404 returned error can't find the container with id cf6193450227f9f4f242042ed4f7f2b320ab85f497b85dd7143e7ce64ecf980e Nov 02 23:25:22.967983 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:22.967953 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6"] Nov 02 23:25:22.969027 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:25:22.968995 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7e8e04_848f_40e6_8445_c3e8bdd8ffea.slice/crio-6a5e2bb00d28ce6573b19d9f5b637abfecbcfd795e584b9ebb3fac7c6c2787bf WatchSource:0}: Error finding container 6a5e2bb00d28ce6573b19d9f5b637abfecbcfd795e584b9ebb3fac7c6c2787bf: Status 404 returned error can't find the container with id 6a5e2bb00d28ce6573b19d9f5b637abfecbcfd795e584b9ebb3fac7c6c2787bf Nov 02 23:25:23.002045 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:23.002012 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" event={"ID":"58fff7c5-ccef-4ee3-a661-300ef85fbfe6","Type":"ContainerStarted","Data":"b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0"} Nov 02 23:25:23.002186 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:23.002104 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" event={"ID":"58fff7c5-ccef-4ee3-a661-300ef85fbfe6","Type":"ContainerStarted","Data":"cf6193450227f9f4f242042ed4f7f2b320ab85f497b85dd7143e7ce64ecf980e"} Nov 02 23:25:23.002186 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:23.002137 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:25:23.003469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:23.003438 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" event={"ID":"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea","Type":"ContainerStarted","Data":"6a5e2bb00d28ce6573b19d9f5b637abfecbcfd795e584b9ebb3fac7c6c2787bf"} Nov 02 23:25:23.114030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:23.113974 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" podStartSLOduration=1.113961009 podStartE2EDuration="1.113961009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:23.113215285 +0000 UTC m=+1916.520527112" watchObservedRunningTime="2026-11-02 23:25:23.113961009 +0000 UTC m=+1916.521272873" Nov 02 23:25:24.011168 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:24.011123 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" event={"ID":"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea","Type":"ContainerStarted","Data":"929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b"} Nov 02 23:25:24.011619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:24.011505 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:25:24.117794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:24.117746 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" podStartSLOduration=2.11773269 podStartE2EDuration="2.11773269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:24.116503513 +0000 UTC m=+1917.523815341" watchObservedRunningTime="2026-11-02 23:25:24.11773269 +0000 UTC m=+1917.525044518" Nov 02 23:25:26.021118 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:25:26.021073 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:25:27.985779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:27.985748 5672 scope.go:122] "RemoveContainer" containerID="cd00a841b8c80b904db717e2bcd70473da5fec3c490f77db6060420e95b29f79" Nov 02 23:25:34.014943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:34.014912 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:25:35.019704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:35.019671 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:25:35.247067 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:35.247025 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx"] Nov 02 23:25:39.471749 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.471715 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw"] Nov 02 23:25:39.477278 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.477259 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:25:39.527835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.527788 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw"] Nov 02 23:25:39.584691 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.584661 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-81c3c9-dockercfg-cx4xn\"" Nov 02 23:25:39.585006 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.584986 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-81c3c9\"" Nov 02 23:25:39.649409 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.649374 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch"] Nov 02 23:25:39.654007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.653986 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:25:39.661493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.661464 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d33def5-1207-4710-8abd-04250c8cc5a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw\" (UID: \"8d33def5-1207-4710-8abd-04250c8cc5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:25:39.661574 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.661534 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl4sj\" (UniqueName: \"kubernetes.io/projected/8d33def5-1207-4710-8abd-04250c8cc5a4-kube-api-access-rl4sj\") pod \"payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw\" (UID: \"8d33def5-1207-4710-8abd-04250c8cc5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:25:39.705194 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.705161 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch"] Nov 02 23:25:39.763024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.762928 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch\" (UID: \"ce8f5119-0c53-41e6-a97c-1984fffa7aa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:25:39.763024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.762993 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjj5l\" (UniqueName: \"kubernetes.io/projected/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-kube-api-access-fjj5l\") pod \"payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch\" (UID: \"ce8f5119-0c53-41e6-a97c-1984fffa7aa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:25:39.763316 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.763243 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d33def5-1207-4710-8abd-04250c8cc5a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw\" (UID: \"8d33def5-1207-4710-8abd-04250c8cc5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:25:39.763383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.763312 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl4sj\" (UniqueName: \"kubernetes.io/projected/8d33def5-1207-4710-8abd-04250c8cc5a4-kube-api-access-rl4sj\") pod \"payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw\" (UID: \"8d33def5-1207-4710-8abd-04250c8cc5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:25:39.764129 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.764105 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d33def5-1207-4710-8abd-04250c8cc5a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw\" (UID: \"8d33def5-1207-4710-8abd-04250c8cc5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:25:39.816797 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.816764 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl4sj\" (UniqueName: \"kubernetes.io/projected/8d33def5-1207-4710-8abd-04250c8cc5a4-kube-api-access-rl4sj\") pod \"payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw\" (UID: \"8d33def5-1207-4710-8abd-04250c8cc5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:25:39.864119 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.864086 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch\" (UID: \"ce8f5119-0c53-41e6-a97c-1984fffa7aa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:25:39.864334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.864160 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjj5l\" (UniqueName: \"kubernetes.io/projected/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-kube-api-access-fjj5l\") pod \"payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch\" (UID: \"ce8f5119-0c53-41e6-a97c-1984fffa7aa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:25:39.864848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.864827 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch\" (UID: \"ce8f5119-0c53-41e6-a97c-1984fffa7aa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:25:39.917098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.917072 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjj5l\" (UniqueName: \"kubernetes.io/projected/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-kube-api-access-fjj5l\") pod \"payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch\" (UID: \"ce8f5119-0c53-41e6-a97c-1984fffa7aa5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:25:39.968560 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:39.968533 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:25:40.090926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:40.090892 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:25:40.247326 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:40.247281 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" podUID="97b25fb9-fb60-4467-ae20-958bb09a5bfa" containerName="payload-pre-processing" containerID="cri-o://34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798" gracePeriod=25 Nov 02 23:25:40.343251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:40.343140 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch"] Nov 02 23:25:40.351223 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:25:40.351188 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8f5119_0c53_41e6_a97c_1984fffa7aa5.slice/crio-123dc306bd80c73d7195a065283b6fcced5c1c8a196be9df98837c542fcc8a92 WatchSource:0}: Error finding container 123dc306bd80c73d7195a065283b6fcced5c1c8a196be9df98837c542fcc8a92: Status 404 returned error can't find the container with id 123dc306bd80c73d7195a065283b6fcced5c1c8a196be9df98837c542fcc8a92 Nov 02 23:25:40.459093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:40.458922 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw"] Nov 02 23:25:40.466270 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:25:40.466195 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d33def5_1207_4710_8abd_04250c8cc5a4.slice/crio-27a8a50316b422b9208089f0c9cf6ae004e0223fb4a9f8dc85c2cff99f32e61b WatchSource:0}: Error finding container 27a8a50316b422b9208089f0c9cf6ae004e0223fb4a9f8dc85c2cff99f32e61b: Status 404 returned error can't find the container with id 27a8a50316b422b9208089f0c9cf6ae004e0223fb4a9f8dc85c2cff99f32e61b Nov 02 23:25:40.522424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:40.522402 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:25:40.677827 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:40.677736 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b25fb9-fb60-4467-ae20-958bb09a5bfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b25fb9-fb60-4467-ae20-958bb09a5bfa-plugins-config-volume\") pod \"97b25fb9-fb60-4467-ae20-958bb09a5bfa\" (UID: \"97b25fb9-fb60-4467-ae20-958bb09a5bfa\") " Nov 02 23:25:40.677989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:40.677855 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b25fb9-fb60-4467-ae20-958bb09a5bfa-kube-api-access-wrqmj\" (UniqueName: \"kubernetes.io/projected/97b25fb9-fb60-4467-ae20-958bb09a5bfa-kube-api-access-wrqmj\") pod \"97b25fb9-fb60-4467-ae20-958bb09a5bfa\" (UID: \"97b25fb9-fb60-4467-ae20-958bb09a5bfa\") " Nov 02 23:25:40.678189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:40.678165 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b25fb9-fb60-4467-ae20-958bb09a5bfa-plugins-config-volume" pod "97b25fb9-fb60-4467-ae20-958bb09a5bfa" (UID: "97b25fb9-fb60-4467-ae20-958bb09a5bfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:25:40.678303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:40.678283 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b25fb9-fb60-4467-ae20-958bb09a5bfa-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:40.679846 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:40.679820 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b25fb9-fb60-4467-ae20-958bb09a5bfa-kube-api-access-wrqmj" pod "97b25fb9-fb60-4467-ae20-958bb09a5bfa" (UID: "97b25fb9-fb60-4467-ae20-958bb09a5bfa"). InnerVolumeSpecName "kube-api-access-wrqmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:25:40.779377 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:40.779340 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrqmj\" (UniqueName: \"kubernetes.io/projected/97b25fb9-fb60-4467-ae20-958bb09a5bfa-kube-api-access-wrqmj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:25:41.128190 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.128135 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" event={"ID":"ce8f5119-0c53-41e6-a97c-1984fffa7aa5","Type":"ContainerStarted","Data":"94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c"} Nov 02 23:25:41.128381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.128216 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" event={"ID":"ce8f5119-0c53-41e6-a97c-1984fffa7aa5","Type":"ContainerStarted","Data":"123dc306bd80c73d7195a065283b6fcced5c1c8a196be9df98837c542fcc8a92"} Nov 02 23:25:41.128381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.128337 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:25:41.129553 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.129531 5672 generic.go:356] "Generic (PLEG): container finished" podID="97b25fb9-fb60-4467-ae20-958bb09a5bfa" containerID="34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798" exitCode=0 Nov 02 23:25:41.129669 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.129567 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" Nov 02 23:25:41.129669 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.129611 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" event={"ID":"97b25fb9-fb60-4467-ae20-958bb09a5bfa","Type":"ContainerDied","Data":"34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798"} Nov 02 23:25:41.129669 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.129648 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx" event={"ID":"97b25fb9-fb60-4467-ae20-958bb09a5bfa","Type":"ContainerDied","Data":"e45ab3aa249ea54e4fcd84b5044c93b263a7cbcd13258935994b10fb497d0cff"} Nov 02 23:25:41.129822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.129674 5672 scope.go:122] "RemoveContainer" containerID="34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798" Nov 02 23:25:41.131408 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.131366 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" event={"ID":"8d33def5-1207-4710-8abd-04250c8cc5a4","Type":"ContainerStarted","Data":"7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048"} Nov 02 23:25:41.131542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.131424 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:25:41.131542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.131450 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" event={"ID":"8d33def5-1207-4710-8abd-04250c8cc5a4","Type":"ContainerStarted","Data":"27a8a50316b422b9208089f0c9cf6ae004e0223fb4a9f8dc85c2cff99f32e61b"} Nov 02 23:25:41.142189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.142171 5672 scope.go:122] "RemoveContainer" containerID="34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798" Nov 02 23:25:41.142485 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:25:41.142466 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798\": container with ID starting with 34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798 not found: ID does not exist" containerID="34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798" Nov 02 23:25:41.142549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.142491 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798"} err="failed to get container status \"34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798\": rpc error: code = NotFound desc = could not find container \"34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798\": container with ID starting with 34d1194b25e510316e7085c5cd5aebd4e0c87423ba0bf4a3a06f323e92cd1798 not found: ID does not exist" Nov 02 23:25:41.236063 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.236024 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" podStartSLOduration=2.236013523 podStartE2EDuration="2.236013523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:41.234317965 +0000 UTC m=+1934.641629788" watchObservedRunningTime="2026-11-02 23:25:41.236013523 +0000 UTC m=+1934.643325352" Nov 02 23:25:41.342045 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.341994 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" podStartSLOduration=2.341981553 podStartE2EDuration="2.341981553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:25:41.340395051 +0000 UTC m=+1934.747706881" watchObservedRunningTime="2026-11-02 23:25:41.341981553 +0000 UTC m=+1934.749293376" Nov 02 23:25:41.446051 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.445956 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx"] Nov 02 23:25:41.451477 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:41.451451 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-981724-5ff47678c-wnvxx"] Nov 02 23:25:42.713169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:42.713128 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b25fb9-fb60-4467-ae20-958bb09a5bfa" path="/var/lib/kubelet/pods/97b25fb9-fb60-4467-ae20-958bb09a5bfa/volumes" Nov 02 23:25:52.141033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:52.140997 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:25:52.141525 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:52.141075 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:25:54.230534 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:25:54.230497 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:25:56.021976 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:25:56.021934 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:26:26.023959 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:26:26.023917 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:26:26.445256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.445169 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb"] Nov 02 23:26:26.445723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.445707 5672 cpu_manager.go:414] "containerMap: removing container" podUID="97b25fb9-fb60-4467-ae20-958bb09a5bfa" containerName="payload-pre-processing" Nov 02 23:26:26.445797 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.445726 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b25fb9-fb60-4467-ae20-958bb09a5bfa" containerName="payload-pre-processing" Nov 02 23:26:26.445866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.445819 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b25fb9-fb60-4467-ae20-958bb09a5bfa" containerName="payload-pre-processing" Nov 02 23:26:26.449060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.449039 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:26:26.498855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.498824 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3bf62f\"" Nov 02 23:26:26.502176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.502131 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb"] Nov 02 23:26:26.514918 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.514887 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbj4x\" (UniqueName: \"kubernetes.io/projected/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-kube-api-access-lbj4x\") pod \"payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb\" (UID: \"f94e7d1e-d9e0-4268-8125-12e0f6e075c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:26:26.515076 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.515058 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb\" (UID: \"f94e7d1e-d9e0-4268-8125-12e0f6e075c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:26:26.548172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.545208 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3bf62f-dockercfg-h2j5g\"" Nov 02 23:26:26.615911 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.615874 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb\" (UID: \"f94e7d1e-d9e0-4268-8125-12e0f6e075c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:26:26.616120 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.615971 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbj4x\" (UniqueName: \"kubernetes.io/projected/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-kube-api-access-lbj4x\") pod \"payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb\" (UID: \"f94e7d1e-d9e0-4268-8125-12e0f6e075c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:26:26.616646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.616626 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb\" (UID: \"f94e7d1e-d9e0-4268-8125-12e0f6e075c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:26:26.669984 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.669953 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbj4x\" (UniqueName: \"kubernetes.io/projected/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-kube-api-access-lbj4x\") pod \"payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb\" (UID: \"f94e7d1e-d9e0-4268-8125-12e0f6e075c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:26:26.812229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.812191 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3bf62f-dockercfg-h2j5g\"" Nov 02 23:26:26.814771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:26.814736 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:26:27.196613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:27.196588 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb"] Nov 02 23:26:27.199163 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:26:27.199122 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94e7d1e_d9e0_4268_8125_12e0f6e075c9.slice/crio-b080501c0f5190265f7fb1c4476f3fba03b90d193d314af3937a0dd7c76595b4 WatchSource:0}: Error finding container b080501c0f5190265f7fb1c4476f3fba03b90d193d314af3937a0dd7c76595b4: Status 404 returned error can't find the container with id b080501c0f5190265f7fb1c4476f3fba03b90d193d314af3937a0dd7c76595b4 Nov 02 23:26:27.446188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:27.446019 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" event={"ID":"f94e7d1e-d9e0-4268-8125-12e0f6e075c9","Type":"ContainerStarted","Data":"f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24"} Nov 02 23:26:27.446188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:27.446090 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" event={"ID":"f94e7d1e-d9e0-4268-8125-12e0f6e075c9","Type":"ContainerStarted","Data":"b080501c0f5190265f7fb1c4476f3fba03b90d193d314af3937a0dd7c76595b4"} Nov 02 23:26:27.446188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:27.446121 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:26:27.552984 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:27.552928 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" podStartSLOduration=1.55291376 podStartE2EDuration="1.55291376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:27.551333412 +0000 UTC m=+1980.958645243" watchObservedRunningTime="2026-11-02 23:26:27.55291376 +0000 UTC m=+1980.960225590" Nov 02 23:26:38.456908 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:38.456837 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:26:45.652579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:26:45.652538 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:26:56.021024 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:26:56.020977 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:27:02.009348 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.009314 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4"] Nov 02 23:27:02.013392 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.013369 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:27:02.109818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.109786 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5ff336\"" Nov 02 23:27:02.112041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.110059 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5ff336-dockercfg-fddff\"" Nov 02 23:27:02.115056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.115023 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4"] Nov 02 23:27:02.147391 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.147353 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2620c879-d7a2-46cf-a94e-6624cc4a291a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4\" (UID: \"2620c879-d7a2-46cf-a94e-6624cc4a291a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:27:02.150075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.150045 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d8c5\" (UniqueName: \"kubernetes.io/projected/2620c879-d7a2-46cf-a94e-6624cc4a291a-kube-api-access-2d8c5\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4\" (UID: \"2620c879-d7a2-46cf-a94e-6624cc4a291a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:27:02.169115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.169084 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h"] Nov 02 23:27:02.173270 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.173243 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:27:02.224102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.224072 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h"] Nov 02 23:27:02.251418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.251383 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13ce135-1fe4-494a-b161-a74545a17d44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h\" (UID: \"c13ce135-1fe4-494a-b161-a74545a17d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:27:02.251635 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.251516 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2620c879-d7a2-46cf-a94e-6624cc4a291a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4\" (UID: \"2620c879-d7a2-46cf-a94e-6624cc4a291a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:27:02.251704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.251659 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8c5\" (UniqueName: \"kubernetes.io/projected/2620c879-d7a2-46cf-a94e-6624cc4a291a-kube-api-access-2d8c5\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4\" (UID: \"2620c879-d7a2-46cf-a94e-6624cc4a291a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:27:02.251761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.251721 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2b8r\" (UniqueName: \"kubernetes.io/projected/c13ce135-1fe4-494a-b161-a74545a17d44-kube-api-access-c2b8r\") pod \"payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h\" (UID: \"c13ce135-1fe4-494a-b161-a74545a17d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:27:02.252913 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.252882 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2620c879-d7a2-46cf-a94e-6624cc4a291a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4\" (UID: \"2620c879-d7a2-46cf-a94e-6624cc4a291a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:27:02.306881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.306805 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8c5\" (UniqueName: \"kubernetes.io/projected/2620c879-d7a2-46cf-a94e-6624cc4a291a-kube-api-access-2d8c5\") pod \"payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4\" (UID: \"2620c879-d7a2-46cf-a94e-6624cc4a291a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:27:02.326270 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.326242 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:27:02.352803 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.352769 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2b8r\" (UniqueName: \"kubernetes.io/projected/c13ce135-1fe4-494a-b161-a74545a17d44-kube-api-access-c2b8r\") pod \"payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h\" (UID: \"c13ce135-1fe4-494a-b161-a74545a17d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:27:02.352932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.352884 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13ce135-1fe4-494a-b161-a74545a17d44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h\" (UID: \"c13ce135-1fe4-494a-b161-a74545a17d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:27:02.353821 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.353792 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13ce135-1fe4-494a-b161-a74545a17d44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h\" (UID: \"c13ce135-1fe4-494a-b161-a74545a17d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:27:02.407622 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.407592 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2b8r\" (UniqueName: \"kubernetes.io/projected/c13ce135-1fe4-494a-b161-a74545a17d44-kube-api-access-c2b8r\") pod \"payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h\" (UID: \"c13ce135-1fe4-494a-b161-a74545a17d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:27:02.486868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.486833 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:27:02.701243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.701218 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4"] Nov 02 23:27:02.702541 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:27:02.702508 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2620c879_d7a2_46cf_a94e_6624cc4a291a.slice/crio-f87c5952738dbd1de510ecfdfb057eb460d43fcbfea4797058a5b1e4d593c576 WatchSource:0}: Error finding container f87c5952738dbd1de510ecfdfb057eb460d43fcbfea4797058a5b1e4d593c576: Status 404 returned error can't find the container with id f87c5952738dbd1de510ecfdfb057eb460d43fcbfea4797058a5b1e4d593c576 Nov 02 23:27:02.854438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:02.854411 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h"] Nov 02 23:27:02.856672 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:27:02.856644 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13ce135_1fe4_494a_b161_a74545a17d44.slice/crio-b702768a6c33dcbe04b91bfa8f0ce3531e5546dc6d58ec18032bd4b00ede0528 WatchSource:0}: Error finding container b702768a6c33dcbe04b91bfa8f0ce3531e5546dc6d58ec18032bd4b00ede0528: Status 404 returned error can't find the container with id b702768a6c33dcbe04b91bfa8f0ce3531e5546dc6d58ec18032bd4b00ede0528 Nov 02 23:27:03.695503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:03.695462 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" event={"ID":"2620c879-d7a2-46cf-a94e-6624cc4a291a","Type":"ContainerStarted","Data":"156134ae71928789b2a439221628db6e4358744a692c6bf02e8a5ef931227382"} Nov 02 23:27:03.695965 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:03.695522 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" event={"ID":"2620c879-d7a2-46cf-a94e-6624cc4a291a","Type":"ContainerStarted","Data":"f87c5952738dbd1de510ecfdfb057eb460d43fcbfea4797058a5b1e4d593c576"} Nov 02 23:27:03.695965 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:03.695609 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:27:03.697215 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:03.697188 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" event={"ID":"c13ce135-1fe4-494a-b161-a74545a17d44","Type":"ContainerStarted","Data":"3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae"} Nov 02 23:27:03.697348 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:03.697231 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" event={"ID":"c13ce135-1fe4-494a-b161-a74545a17d44","Type":"ContainerStarted","Data":"b702768a6c33dcbe04b91bfa8f0ce3531e5546dc6d58ec18032bd4b00ede0528"} Nov 02 23:27:03.697348 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:03.697256 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:27:03.804253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:03.804203 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" podStartSLOduration=2.804188066 podStartE2EDuration="2.804188066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:03.803305634 +0000 UTC m=+2017.210617465" watchObservedRunningTime="2026-11-02 23:27:03.804188066 +0000 UTC m=+2017.211499915" Nov 02 23:27:03.912071 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:03.912017 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" podStartSLOduration=1.912002439 podStartE2EDuration="1.912002439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:03.909750755 +0000 UTC m=+2017.317062584" watchObservedRunningTime="2026-11-02 23:27:03.912002439 +0000 UTC m=+2017.319314269" Nov 02 23:27:14.712867 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:14.712840 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:27:14.713343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:14.712900 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:27:24.197974 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:24.197939 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf"] Nov 02 23:27:24.198436 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:24.198142 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" containerID="cri-o://0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4" gracePeriod=30 Nov 02 23:27:26.021175 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:27:26.021111 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:27:28.437219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:28.436966 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:27:28.437219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:28.437026 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:27:29.464357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.464328 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:27:29.567256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.567218 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-token\" (UniqueName: \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-token\") pod \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " Nov 02 23:27:29.567256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.567258 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-envoy\") pod \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " Nov 02 23:27:29.567487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.567288 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-podinfo\") pod \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " Nov 02 23:27:29.567487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.567441 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istiod-ca-cert\") pod \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " Nov 02 23:27:29.567603 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.567540 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-credential-socket\") pod \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " Nov 02 23:27:29.567603 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.567588 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-data\") pod \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " Nov 02 23:27:29.567708 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.567628 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-certs\") pod \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " Nov 02 23:27:29.567840 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.567794 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-socket\") pod \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " Nov 02 23:27:29.567916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.567832 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-credential-socket" pod "15b474fc-c267-4b2d-b2f0-72340c7ecc69" (UID: "15b474fc-c267-4b2d-b2f0-72340c7ecc69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:27:29.567916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.567839 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-kube-api-access-4tzbk\" (UniqueName: \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-kube-api-access-4tzbk\") pod \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\" (UID: \"15b474fc-c267-4b2d-b2f0-72340c7ecc69\") " Nov 02 23:27:29.567916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.567862 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istiod-ca-cert" pod "15b474fc-c267-4b2d-b2f0-72340c7ecc69" (UID: "15b474fc-c267-4b2d-b2f0-72340c7ecc69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:27:29.568548 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.568346 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-data" pod "15b474fc-c267-4b2d-b2f0-72340c7ecc69" (UID: "15b474fc-c267-4b2d-b2f0-72340c7ecc69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:27:29.568548 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.568436 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-certs" pod "15b474fc-c267-4b2d-b2f0-72340c7ecc69" (UID: "15b474fc-c267-4b2d-b2f0-72340c7ecc69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:27:29.568548 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.568493 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:29.568548 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.568512 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:29.568548 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.568526 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:29.568882 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.568737 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-socket" pod "15b474fc-c267-4b2d-b2f0-72340c7ecc69" (UID: "15b474fc-c267-4b2d-b2f0-72340c7ecc69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:27:29.569772 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.569747 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-podinfo" pod "15b474fc-c267-4b2d-b2f0-72340c7ecc69" (UID: "15b474fc-c267-4b2d-b2f0-72340c7ecc69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:27:29.570016 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.569989 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-token" pod "15b474fc-c267-4b2d-b2f0-72340c7ecc69" (UID: "15b474fc-c267-4b2d-b2f0-72340c7ecc69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:27:29.570139 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.570041 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-envoy" pod "15b474fc-c267-4b2d-b2f0-72340c7ecc69" (UID: "15b474fc-c267-4b2d-b2f0-72340c7ecc69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:27:29.570412 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.570390 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-kube-api-access-4tzbk" pod "15b474fc-c267-4b2d-b2f0-72340c7ecc69" (UID: "15b474fc-c267-4b2d-b2f0-72340c7ecc69"). InnerVolumeSpecName "kube-api-access-4tzbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:27:29.669976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.669846 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:29.669976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.669883 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:29.669976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.669898 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b474fc-c267-4b2d-b2f0-72340c7ecc69-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:29.669976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.669914 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:29.669976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.669928 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b474fc-c267-4b2d-b2f0-72340c7ecc69-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:29.669976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.669942 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tzbk\" (UniqueName: \"kubernetes.io/projected/15b474fc-c267-4b2d-b2f0-72340c7ecc69-kube-api-access-4tzbk\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:29.798133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.798093 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh"] Nov 02 23:27:29.863995 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.863898 5672 generic.go:356] "Generic (PLEG): container finished" podID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerID="0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4" exitCode=0 Nov 02 23:27:29.863995 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.863954 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" Nov 02 23:27:29.863995 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.863973 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" event={"ID":"15b474fc-c267-4b2d-b2f0-72340c7ecc69","Type":"ContainerDied","Data":"0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4"} Nov 02 23:27:29.864340 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.864029 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf" event={"ID":"15b474fc-c267-4b2d-b2f0-72340c7ecc69","Type":"ContainerDied","Data":"d35ddd478655d130cbcd00cf441d448501540382175bc2e41c661f1bd48a0a9e"} Nov 02 23:27:29.864340 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.864052 5672 scope.go:122] "RemoveContainer" containerID="0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4" Nov 02 23:27:29.876292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.876270 5672 scope.go:122] "RemoveContainer" containerID="0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4" Nov 02 23:27:29.876569 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:27:29.876545 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4\": container with ID starting with 0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4 not found: ID does not exist" containerID="0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4" Nov 02 23:27:29.876661 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.876579 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4"} err="failed to get container status \"0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4\": rpc error: code = NotFound desc = could not find container \"0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4\": container with ID starting with 0fd0c57fefe9fe41fc6eba1f70f4ca3a975a12170786eebc9390d1c0120782b4 not found: ID does not exist" Nov 02 23:27:29.899297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:29.899266 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp"] Nov 02 23:27:30.012096 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:30.012058 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf"] Nov 02 23:27:30.017226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:30.017198 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-4rwcf"] Nov 02 23:27:30.713773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:30.713741 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" path="/var/lib/kubelet/pods/15b474fc-c267-4b2d-b2f0-72340c7ecc69/volumes" Nov 02 23:27:31.309802 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:31.309770 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:27:34.799465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:34.799403 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" podUID="2ddc0a44-811e-40dc-ac64-57a75bf439be" containerName="payload-processing" containerID="cri-o://ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876" gracePeriod=25 Nov 02 23:27:34.899878 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:34.899844 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" podUID="c267ca54-0618-4561-9d5f-95c3c7cb655b" containerName="payload-pre-processing" containerID="cri-o://62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42" gracePeriod=25 Nov 02 23:27:35.074079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.074048 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:27:35.170771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.170738 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:27:35.235784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.235738 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ddc0a44-811e-40dc-ac64-57a75bf439be-kube-api-access-r45fx\" (UniqueName: \"kubernetes.io/projected/2ddc0a44-811e-40dc-ac64-57a75bf439be-kube-api-access-r45fx\") pod \"2ddc0a44-811e-40dc-ac64-57a75bf439be\" (UID: \"2ddc0a44-811e-40dc-ac64-57a75bf439be\") " Nov 02 23:27:35.236035 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.236016 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ddc0a44-811e-40dc-ac64-57a75bf439be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ddc0a44-811e-40dc-ac64-57a75bf439be-plugins-config-volume\") pod \"2ddc0a44-811e-40dc-ac64-57a75bf439be\" (UID: \"2ddc0a44-811e-40dc-ac64-57a75bf439be\") " Nov 02 23:27:35.236465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.236418 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ddc0a44-811e-40dc-ac64-57a75bf439be-plugins-config-volume" pod "2ddc0a44-811e-40dc-ac64-57a75bf439be" (UID: "2ddc0a44-811e-40dc-ac64-57a75bf439be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:27:35.238629 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.238605 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ddc0a44-811e-40dc-ac64-57a75bf439be-kube-api-access-r45fx" pod "2ddc0a44-811e-40dc-ac64-57a75bf439be" (UID: "2ddc0a44-811e-40dc-ac64-57a75bf439be"). InnerVolumeSpecName "kube-api-access-r45fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:27:35.337203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.337084 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c267ca54-0618-4561-9d5f-95c3c7cb655b-kube-api-access-rpbtr\" (UniqueName: \"kubernetes.io/projected/c267ca54-0618-4561-9d5f-95c3c7cb655b-kube-api-access-rpbtr\") pod \"c267ca54-0618-4561-9d5f-95c3c7cb655b\" (UID: \"c267ca54-0618-4561-9d5f-95c3c7cb655b\") " Nov 02 23:27:35.337203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.337142 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c267ca54-0618-4561-9d5f-95c3c7cb655b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c267ca54-0618-4561-9d5f-95c3c7cb655b-plugins-config-volume\") pod \"c267ca54-0618-4561-9d5f-95c3c7cb655b\" (UID: \"c267ca54-0618-4561-9d5f-95c3c7cb655b\") " Nov 02 23:27:35.337665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.337637 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c267ca54-0618-4561-9d5f-95c3c7cb655b-plugins-config-volume" pod "c267ca54-0618-4561-9d5f-95c3c7cb655b" (UID: "c267ca54-0618-4561-9d5f-95c3c7cb655b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:27:35.337800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.337781 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c267ca54-0618-4561-9d5f-95c3c7cb655b-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:35.337859 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.337807 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ddc0a44-811e-40dc-ac64-57a75bf439be-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:35.337859 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.337823 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r45fx\" (UniqueName: \"kubernetes.io/projected/2ddc0a44-811e-40dc-ac64-57a75bf439be-kube-api-access-r45fx\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:35.339886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.339854 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c267ca54-0618-4561-9d5f-95c3c7cb655b-kube-api-access-rpbtr" pod "c267ca54-0618-4561-9d5f-95c3c7cb655b" (UID: "c267ca54-0618-4561-9d5f-95c3c7cb655b"). InnerVolumeSpecName "kube-api-access-rpbtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:27:35.439129 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.439097 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpbtr\" (UniqueName: \"kubernetes.io/projected/c267ca54-0618-4561-9d5f-95c3c7cb655b-kube-api-access-rpbtr\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:35.907972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.907935 5672 generic.go:356] "Generic (PLEG): container finished" podID="2ddc0a44-811e-40dc-ac64-57a75bf439be" containerID="ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876" exitCode=0 Nov 02 23:27:35.908464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.907982 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" Nov 02 23:27:35.908464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.908011 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" event={"ID":"2ddc0a44-811e-40dc-ac64-57a75bf439be","Type":"ContainerDied","Data":"ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876"} Nov 02 23:27:35.908464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.908058 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh" event={"ID":"2ddc0a44-811e-40dc-ac64-57a75bf439be","Type":"ContainerDied","Data":"b2417fd747689c46b174c1c118ffc85b086c5cc971eb3576e9a12065d6b18216"} Nov 02 23:27:35.908464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.908079 5672 scope.go:122] "RemoveContainer" containerID="ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876" Nov 02 23:27:35.909706 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.909684 5672 generic.go:356] "Generic (PLEG): container finished" podID="c267ca54-0618-4561-9d5f-95c3c7cb655b" containerID="62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42" exitCode=0 Nov 02 23:27:35.909818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.909740 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" Nov 02 23:27:35.909818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.909765 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" event={"ID":"c267ca54-0618-4561-9d5f-95c3c7cb655b","Type":"ContainerDied","Data":"62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42"} Nov 02 23:27:35.909818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.909808 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp" event={"ID":"c267ca54-0618-4561-9d5f-95c3c7cb655b","Type":"ContainerDied","Data":"ed5f8b27ee2433a171af1d75f2033eec8cb824c16ff6cdf7225ccda60b65446b"} Nov 02 23:27:35.925416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.925389 5672 scope.go:122] "RemoveContainer" containerID="ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876" Nov 02 23:27:35.925989 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:27:35.925956 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876\": container with ID starting with ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876 not found: ID does not exist" containerID="ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876" Nov 02 23:27:35.926143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.926118 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876"} err="failed to get container status \"ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876\": rpc error: code = NotFound desc = could not find container \"ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876\": container with ID starting with ae5d181b757a2735df48240be02b5ce260fcb7cc738a0a6dab6c8e1f3a863876 not found: ID does not exist" Nov 02 23:27:35.926285 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.926272 5672 scope.go:122] "RemoveContainer" containerID="62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42" Nov 02 23:27:35.936855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.936824 5672 scope.go:122] "RemoveContainer" containerID="62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42" Nov 02 23:27:35.938190 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:27:35.938124 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42\": container with ID starting with 62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42 not found: ID does not exist" containerID="62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42" Nov 02 23:27:35.938539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:35.938436 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42"} err="failed to get container status \"62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42\": rpc error: code = NotFound desc = could not find container \"62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42\": container with ID starting with 62d0691d867e85a58d545460079ebefdb0dc49920838b42c7003029dd4118b42 not found: ID does not exist" Nov 02 23:27:36.061707 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:36.061674 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh"] Nov 02 23:27:36.108015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:36.107947 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b57963-5d7bb67fb7-98sxh"] Nov 02 23:27:36.167090 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:36.167003 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp"] Nov 02 23:27:36.170756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:36.170727 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b57963-7c48fb6567-46hbp"] Nov 02 23:27:36.473226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:36.473191 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6"] Nov 02 23:27:36.575478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:36.575449 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld"] Nov 02 23:27:36.712785 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:36.712754 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ddc0a44-811e-40dc-ac64-57a75bf439be" path="/var/lib/kubelet/pods/2ddc0a44-811e-40dc-ac64-57a75bf439be/volumes" Nov 02 23:27:36.713221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:36.713207 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c267ca54-0618-4561-9d5f-95c3c7cb655b" path="/var/lib/kubelet/pods/c267ca54-0618-4561-9d5f-95c3c7cb655b/volumes" Nov 02 23:27:41.473820 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.473780 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" podUID="0c7e8e04-848f-40e6-8445-c3e8bdd8ffea" containerName="payload-processing" containerID="cri-o://929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b" gracePeriod=25 Nov 02 23:27:41.576687 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.576653 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" podUID="58fff7c5-ccef-4ee3-a661-300ef85fbfe6" containerName="payload-pre-processing" containerID="cri-o://b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0" gracePeriod=25 Nov 02 23:27:41.734364 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.734295 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:27:41.810698 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.810664 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-plugins-config-volume\") pod \"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea\" (UID: \"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea\") " Nov 02 23:27:41.810971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.810774 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-kube-api-access-m4vxt\" (UniqueName: \"kubernetes.io/projected/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-kube-api-access-m4vxt\") pod \"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea\" (UID: \"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea\") " Nov 02 23:27:41.811097 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.811067 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-plugins-config-volume" pod "0c7e8e04-848f-40e6-8445-c3e8bdd8ffea" (UID: "0c7e8e04-848f-40e6-8445-c3e8bdd8ffea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:27:41.811301 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.811283 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:41.813999 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.813968 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-kube-api-access-m4vxt" pod "0c7e8e04-848f-40e6-8445-c3e8bdd8ffea" (UID: "0c7e8e04-848f-40e6-8445-c3e8bdd8ffea"). InnerVolumeSpecName "kube-api-access-m4vxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:27:41.840597 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.840576 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:27:41.911653 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.911625 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-plugins-config-volume\") pod \"58fff7c5-ccef-4ee3-a661-300ef85fbfe6\" (UID: \"58fff7c5-ccef-4ee3-a661-300ef85fbfe6\") " Nov 02 23:27:41.911838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.911744 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-kube-api-access-w27m6\" (UniqueName: \"kubernetes.io/projected/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-kube-api-access-w27m6\") pod \"58fff7c5-ccef-4ee3-a661-300ef85fbfe6\" (UID: \"58fff7c5-ccef-4ee3-a661-300ef85fbfe6\") " Nov 02 23:27:41.912112 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.912087 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-plugins-config-volume" pod "58fff7c5-ccef-4ee3-a661-300ef85fbfe6" (UID: "58fff7c5-ccef-4ee3-a661-300ef85fbfe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:27:41.912163 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.912122 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4vxt\" (UniqueName: \"kubernetes.io/projected/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea-kube-api-access-m4vxt\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:41.913896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.913875 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-kube-api-access-w27m6" pod "58fff7c5-ccef-4ee3-a661-300ef85fbfe6" (UID: "58fff7c5-ccef-4ee3-a661-300ef85fbfe6"). InnerVolumeSpecName "kube-api-access-w27m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:27:41.953800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.953768 5672 generic.go:356] "Generic (PLEG): container finished" podID="0c7e8e04-848f-40e6-8445-c3e8bdd8ffea" containerID="929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b" exitCode=0 Nov 02 23:27:41.953979 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.953943 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" Nov 02 23:27:41.954044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.953973 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" event={"ID":"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea","Type":"ContainerDied","Data":"929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b"} Nov 02 23:27:41.954102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.954059 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6" event={"ID":"0c7e8e04-848f-40e6-8445-c3e8bdd8ffea","Type":"ContainerDied","Data":"6a5e2bb00d28ce6573b19d9f5b637abfecbcfd795e584b9ebb3fac7c6c2787bf"} Nov 02 23:27:41.954193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.954108 5672 scope.go:122] "RemoveContainer" containerID="929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b" Nov 02 23:27:41.956074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.956049 5672 generic.go:356] "Generic (PLEG): container finished" podID="58fff7c5-ccef-4ee3-a661-300ef85fbfe6" containerID="b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0" exitCode=0 Nov 02 23:27:41.956267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.956085 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" event={"ID":"58fff7c5-ccef-4ee3-a661-300ef85fbfe6","Type":"ContainerDied","Data":"b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0"} Nov 02 23:27:41.956267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.956123 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" event={"ID":"58fff7c5-ccef-4ee3-a661-300ef85fbfe6","Type":"ContainerDied","Data":"cf6193450227f9f4f242042ed4f7f2b320ab85f497b85dd7143e7ce64ecf980e"} Nov 02 23:27:41.956267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.956235 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld" Nov 02 23:27:41.966831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.966744 5672 scope.go:122] "RemoveContainer" containerID="929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b" Nov 02 23:27:41.967097 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:27:41.967068 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b\": container with ID starting with 929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b not found: ID does not exist" containerID="929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b" Nov 02 23:27:41.967208 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.967108 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b"} err="failed to get container status \"929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b\": rpc error: code = NotFound desc = could not find container \"929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b\": container with ID starting with 929ca055db9ce777d3a0d0d65ff084e986ee95565ee9d783d7fc5b38355de31b not found: ID does not exist" Nov 02 23:27:41.967208 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.967132 5672 scope.go:122] "RemoveContainer" containerID="b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0" Nov 02 23:27:41.977029 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.977013 5672 scope.go:122] "RemoveContainer" containerID="b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0" Nov 02 23:27:41.977337 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:27:41.977316 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0\": container with ID starting with b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0 not found: ID does not exist" containerID="b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0" Nov 02 23:27:41.977406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:41.977342 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0"} err="failed to get container status \"b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0\": rpc error: code = NotFound desc = could not find container \"b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0\": container with ID starting with b698d36ac00a20382d52411dd2e183eac7fc1443a47d7c47cabe9c1439df53d0 not found: ID does not exist" Nov 02 23:27:42.013181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:42.013106 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:42.013181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:42.013128 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w27m6\" (UniqueName: \"kubernetes.io/projected/58fff7c5-ccef-4ee3-a661-300ef85fbfe6-kube-api-access-w27m6\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:27:42.106138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:42.106110 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6"] Nov 02 23:27:42.109547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:42.109524 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fdfca-bbc7bb7cc-fwxw6"] Nov 02 23:27:42.210692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:42.210657 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld"] Nov 02 23:27:42.213932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:42.213909 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fdfca-6b449f6f99-6bhld"] Nov 02 23:27:42.713379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:42.713347 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c7e8e04-848f-40e6-8445-c3e8bdd8ffea" path="/var/lib/kubelet/pods/0c7e8e04-848f-40e6-8445-c3e8bdd8ffea/volumes" Nov 02 23:27:42.713750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:42.713715 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58fff7c5-ccef-4ee3-a661-300ef85fbfe6" path="/var/lib/kubelet/pods/58fff7c5-ccef-4ee3-a661-300ef85fbfe6/volumes" Nov 02 23:27:56.041197 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:27:56.040705 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:27:59.121509 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.121473 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj"] Nov 02 23:27:59.122031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122010 5672 cpu_manager.go:414] "containerMap: removing container" podUID="58fff7c5-ccef-4ee3-a661-300ef85fbfe6" containerName="payload-pre-processing" Nov 02 23:27:59.122031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122028 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fff7c5-ccef-4ee3-a661-300ef85fbfe6" containerName="payload-pre-processing" Nov 02 23:27:59.122271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122065 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c267ca54-0618-4561-9d5f-95c3c7cb655b" containerName="payload-pre-processing" Nov 02 23:27:59.122271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122072 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c267ca54-0618-4561-9d5f-95c3c7cb655b" containerName="payload-pre-processing" Nov 02 23:27:59.122271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122082 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0c7e8e04-848f-40e6-8445-c3e8bdd8ffea" containerName="payload-processing" Nov 02 23:27:59.122271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122089 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7e8e04-848f-40e6-8445-c3e8bdd8ffea" containerName="payload-processing" Nov 02 23:27:59.122271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122128 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2ddc0a44-811e-40dc-ac64-57a75bf439be" containerName="payload-processing" Nov 02 23:27:59.122271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122135 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ddc0a44-811e-40dc-ac64-57a75bf439be" containerName="payload-processing" Nov 02 23:27:59.122271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122169 5672 cpu_manager.go:414] "containerMap: removing container" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" Nov 02 23:27:59.122271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122176 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" Nov 02 23:27:59.122665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122315 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b474fc-c267-4b2d-b2f0-72340c7ecc69" containerName="istio-proxy" Nov 02 23:27:59.122665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122331 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fff7c5-ccef-4ee3-a661-300ef85fbfe6" containerName="payload-pre-processing" Nov 02 23:27:59.122665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122346 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c267ca54-0618-4561-9d5f-95c3c7cb655b" containerName="payload-pre-processing" Nov 02 23:27:59.122665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122357 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7e8e04-848f-40e6-8445-c3e8bdd8ffea" containerName="payload-processing" Nov 02 23:27:59.122665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.122373 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ddc0a44-811e-40dc-ac64-57a75bf439be" containerName="payload-processing" Nov 02 23:27:59.126140 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.126118 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.179536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.178121 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-1c14e6-openshift-default-dockercfg-zsj5h\"" Nov 02 23:27:59.218217 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.218175 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-credential-socket\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.218431 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.218243 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7mch\" (UniqueName: \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-kube-api-access-d7mch\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.218431 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.218335 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-certs\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.218431 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.218376 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-podinfo\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.218431 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.218398 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istiod-ca-cert\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.218716 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.218433 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-socket\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.218716 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.218454 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-envoy\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.218716 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.218474 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-token\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.218716 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.218523 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-data\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.227293 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.227257 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj"] Nov 02 23:27:59.319111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.319069 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-socket\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.319111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.319109 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-envoy\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.319358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.319129 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-token\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.319358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.319209 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-data\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.319358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.319293 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-credential-socket\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.319358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.319318 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7mch\" (UniqueName: \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-kube-api-access-d7mch\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.319549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.319400 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-certs\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.319549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.319438 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-podinfo\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.319549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.319467 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istiod-ca-cert\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.319908 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.319874 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-socket\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.320050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.319968 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-credential-socket\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.320050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.320017 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-data\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.320178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.320071 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-certs\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.320510 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.320488 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istiod-ca-cert\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.321577 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.321557 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-envoy\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.322107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.322078 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-podinfo\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.372692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.372618 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-token\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.372841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.372810 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7mch\" (UniqueName: \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-kube-api-access-d7mch\") pod \"e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.440060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.440025 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:27:59.861013 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.860985 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj"] Nov 02 23:27:59.863298 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:27:59.863261 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8bd222_7c29_4a78_8ab3_a83cb9ed076f.slice/crio-b9dec5228c9f1d505f9b9c3bbb0a36f8b77bf5d5d74757ce0b2a00be91dd63e0 WatchSource:0}: Error finding container b9dec5228c9f1d505f9b9c3bbb0a36f8b77bf5d5d74757ce0b2a00be91dd63e0: Status 404 returned error can't find the container with id b9dec5228c9f1d505f9b9c3bbb0a36f8b77bf5d5d74757ce0b2a00be91dd63e0 Nov 02 23:27:59.866060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.866024 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:27:59.866198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.866129 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:27:59.866284 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:27:59.866206 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:28:00.079205 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:00.079141 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" event={"ID":"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f","Type":"ContainerStarted","Data":"f12c43a7c27fb401fa0cfc1a62126cc5c8dc8365281ccdca6bf71f6c831c95c5"} Nov 02 23:28:00.079371 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:00.079264 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" event={"ID":"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f","Type":"ContainerStarted","Data":"b9dec5228c9f1d505f9b9c3bbb0a36f8b77bf5d5d74757ce0b2a00be91dd63e0"} Nov 02 23:28:00.189674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:00.189400 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" podStartSLOduration=1.189382452 podStartE2EDuration="1.189382452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:28:00.188424052 +0000 UTC m=+2073.595735883" watchObservedRunningTime="2026-11-02 23:28:00.189382452 +0000 UTC m=+2073.596694283" Nov 02 23:28:00.440600 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:00.440500 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:28:00.447586 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:00.447552 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:28:01.085325 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:01.085292 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:28:01.086229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:01.086210 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:28:08.419136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.419058 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b"] Nov 02 23:28:08.423394 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.423372 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:28:08.485165 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.484505 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1c14e6\"" Nov 02 23:28:08.531267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.531221 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-1c14e6-dockercfg-dhswl\"" Nov 02 23:28:08.531667 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.531638 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjmtv\" (UniqueName: \"kubernetes.io/projected/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-kube-api-access-wjmtv\") pod \"payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b\" (UID: \"8b8fac42-a418-4dfe-aad2-1e54e723a8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:28:08.531808 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.531792 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b\" (UID: \"8b8fac42-a418-4dfe-aad2-1e54e723a8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:28:08.533879 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.533854 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b"] Nov 02 23:28:08.590144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.590114 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz"] Nov 02 23:28:08.594575 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.594551 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:28:08.632849 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.632817 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b\" (UID: \"8b8fac42-a418-4dfe-aad2-1e54e723a8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:28:08.633004 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.632933 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjmtv\" (UniqueName: \"kubernetes.io/projected/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-kube-api-access-wjmtv\") pod \"payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b\" (UID: \"8b8fac42-a418-4dfe-aad2-1e54e723a8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:28:08.633486 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.633467 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b\" (UID: \"8b8fac42-a418-4dfe-aad2-1e54e723a8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:28:08.647317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.647286 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz"] Nov 02 23:28:08.686831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.686767 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjmtv\" (UniqueName: \"kubernetes.io/projected/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-kube-api-access-wjmtv\") pod \"payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b\" (UID: \"8b8fac42-a418-4dfe-aad2-1e54e723a8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:28:08.733717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.733667 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9hx\" (UniqueName: \"kubernetes.io/projected/5f3c779c-88f5-4a05-b467-aaa0307d0f56-kube-api-access-mp9hx\") pod \"payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz\" (UID: \"5f3c779c-88f5-4a05-b467-aaa0307d0f56\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:28:08.733895 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.733814 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3c779c-88f5-4a05-b467-aaa0307d0f56-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz\" (UID: \"5f3c779c-88f5-4a05-b467-aaa0307d0f56\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:28:08.738424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.738403 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:28:08.837580 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.837545 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9hx\" (UniqueName: \"kubernetes.io/projected/5f3c779c-88f5-4a05-b467-aaa0307d0f56-kube-api-access-mp9hx\") pod \"payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz\" (UID: \"5f3c779c-88f5-4a05-b467-aaa0307d0f56\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:28:08.837754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.837647 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3c779c-88f5-4a05-b467-aaa0307d0f56-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz\" (UID: \"5f3c779c-88f5-4a05-b467-aaa0307d0f56\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:28:08.838408 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.838383 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3c779c-88f5-4a05-b467-aaa0307d0f56-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz\" (UID: \"5f3c779c-88f5-4a05-b467-aaa0307d0f56\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:28:08.892251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.892222 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9hx\" (UniqueName: \"kubernetes.io/projected/5f3c779c-88f5-4a05-b467-aaa0307d0f56-kube-api-access-mp9hx\") pod \"payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz\" (UID: \"5f3c779c-88f5-4a05-b467-aaa0307d0f56\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:28:08.907745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:08.907710 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:28:09.105935 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:09.105911 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b"] Nov 02 23:28:09.108420 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:28:09.108388 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8fac42_a418_4dfe_aad2_1e54e723a8e0.slice/crio-238746e72b78d2e08cbcd95fdd82743e31aeec6b14ea2c11e4afa647c0cbd625 WatchSource:0}: Error finding container 238746e72b78d2e08cbcd95fdd82743e31aeec6b14ea2c11e4afa647c0cbd625: Status 404 returned error can't find the container with id 238746e72b78d2e08cbcd95fdd82743e31aeec6b14ea2c11e4afa647c0cbd625 Nov 02 23:28:09.141039 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:09.141010 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" event={"ID":"8b8fac42-a418-4dfe-aad2-1e54e723a8e0","Type":"ContainerStarted","Data":"238746e72b78d2e08cbcd95fdd82743e31aeec6b14ea2c11e4afa647c0cbd625"} Nov 02 23:28:09.283677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:09.283650 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz"] Nov 02 23:28:09.285603 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:28:09.285558 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3c779c_88f5_4a05_b467_aaa0307d0f56.slice/crio-987de021ecc1b9e9686f9626d3a92273308ba4cc33ab4dcd605932d370f7a118 WatchSource:0}: Error finding container 987de021ecc1b9e9686f9626d3a92273308ba4cc33ab4dcd605932d370f7a118: Status 404 returned error can't find the container with id 987de021ecc1b9e9686f9626d3a92273308ba4cc33ab4dcd605932d370f7a118 Nov 02 23:28:09.791701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:09.791660 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws"] Nov 02 23:28:09.792138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:09.791874 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerName="istio-proxy" containerID="cri-o://29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a" gracePeriod=30 Nov 02 23:28:10.154639 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:10.154540 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" event={"ID":"8b8fac42-a418-4dfe-aad2-1e54e723a8e0","Type":"ContainerStarted","Data":"ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc"} Nov 02 23:28:10.154919 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:10.154896 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:28:10.158009 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:10.157976 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" event={"ID":"5f3c779c-88f5-4a05-b467-aaa0307d0f56","Type":"ContainerStarted","Data":"bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f"} Nov 02 23:28:10.158171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:10.158034 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" event={"ID":"5f3c779c-88f5-4a05-b467-aaa0307d0f56","Type":"ContainerStarted","Data":"987de021ecc1b9e9686f9626d3a92273308ba4cc33ab4dcd605932d370f7a118"} Nov 02 23:28:10.158282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:10.158262 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:28:10.263319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:10.261813 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" podStartSLOduration=2.261799292 podStartE2EDuration="2.261799292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:28:10.260452861 +0000 UTC m=+2083.667764691" watchObservedRunningTime="2026-11-02 23:28:10.261799292 +0000 UTC m=+2083.669111121" Nov 02 23:28:10.367339 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:10.367276 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" podStartSLOduration=2.367257037 podStartE2EDuration="2.367257037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:28:10.366459719 +0000 UTC m=+2083.773771549" watchObservedRunningTime="2026-11-02 23:28:10.367257037 +0000 UTC m=+2083.774568868" Nov 02 23:28:13.376449 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:13.376412 5672 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:28:13.376847 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:13.376481 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:28:15.063761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.063696 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:28:15.117636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.117606 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8581a625-fd4b-47d0-9ac1-08f312407c1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8581a625-fd4b-47d0-9ac1-08f312407c1b-istiod-ca-cert\") pod \"8581a625-fd4b-47d0-9ac1-08f312407c1b\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " Nov 02 23:28:15.117636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.117647 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-certs\") pod \"8581a625-fd4b-47d0-9ac1-08f312407c1b\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " Nov 02 23:28:15.117903 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.117665 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-kube-api-access-cgqhp\" (UniqueName: \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-kube-api-access-cgqhp\") pod \"8581a625-fd4b-47d0-9ac1-08f312407c1b\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " Nov 02 23:28:15.117903 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.117700 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-envoy\") pod \"8581a625-fd4b-47d0-9ac1-08f312407c1b\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " Nov 02 23:28:15.117903 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.117823 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-podinfo\") pod \"8581a625-fd4b-47d0-9ac1-08f312407c1b\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " Nov 02 23:28:15.117903 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.117896 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-socket\") pod \"8581a625-fd4b-47d0-9ac1-08f312407c1b\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " Nov 02 23:28:15.118113 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.117935 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-token\" (UniqueName: \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-token\") pod \"8581a625-fd4b-47d0-9ac1-08f312407c1b\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " Nov 02 23:28:15.118113 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.117963 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-data\") pod \"8581a625-fd4b-47d0-9ac1-08f312407c1b\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " Nov 02 23:28:15.118113 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.117985 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-certs" pod "8581a625-fd4b-47d0-9ac1-08f312407c1b" (UID: "8581a625-fd4b-47d0-9ac1-08f312407c1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:15.118113 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.118087 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-credential-socket\") pod \"8581a625-fd4b-47d0-9ac1-08f312407c1b\" (UID: \"8581a625-fd4b-47d0-9ac1-08f312407c1b\") " Nov 02 23:28:15.118370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.118171 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8581a625-fd4b-47d0-9ac1-08f312407c1b-istiod-ca-cert" pod "8581a625-fd4b-47d0-9ac1-08f312407c1b" (UID: "8581a625-fd4b-47d0-9ac1-08f312407c1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:15.118430 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.118413 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-data" pod "8581a625-fd4b-47d0-9ac1-08f312407c1b" (UID: "8581a625-fd4b-47d0-9ac1-08f312407c1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:15.118507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.118480 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-socket" pod "8581a625-fd4b-47d0-9ac1-08f312407c1b" (UID: "8581a625-fd4b-47d0-9ac1-08f312407c1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:15.118628 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.118606 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-credential-socket" pod "8581a625-fd4b-47d0-9ac1-08f312407c1b" (UID: "8581a625-fd4b-47d0-9ac1-08f312407c1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:15.118748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.118729 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.118812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.118751 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.118812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.118762 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.118812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.118771 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.118812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.118781 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8581a625-fd4b-47d0-9ac1-08f312407c1b-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.120222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.120191 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-kube-api-access-cgqhp" pod "8581a625-fd4b-47d0-9ac1-08f312407c1b" (UID: "8581a625-fd4b-47d0-9ac1-08f312407c1b"). InnerVolumeSpecName "kube-api-access-cgqhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:15.120327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.120285 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-token" pod "8581a625-fd4b-47d0-9ac1-08f312407c1b" (UID: "8581a625-fd4b-47d0-9ac1-08f312407c1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:15.120428 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.120410 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-podinfo" pod "8581a625-fd4b-47d0-9ac1-08f312407c1b" (UID: "8581a625-fd4b-47d0-9ac1-08f312407c1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:28:15.120534 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.120516 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-envoy" pod "8581a625-fd4b-47d0-9ac1-08f312407c1b" (UID: "8581a625-fd4b-47d0-9ac1-08f312407c1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:15.212849 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.212816 5672 generic.go:356] "Generic (PLEG): container finished" podID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerID="29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a" exitCode=0 Nov 02 23:28:15.213030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.212866 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" Nov 02 23:28:15.213030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.212911 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" event={"ID":"8581a625-fd4b-47d0-9ac1-08f312407c1b","Type":"ContainerDied","Data":"29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a"} Nov 02 23:28:15.213030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.212964 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws" event={"ID":"8581a625-fd4b-47d0-9ac1-08f312407c1b","Type":"ContainerDied","Data":"2f2569980136b35907ebdd599dbc70d136cd95b54a2bde3545f20a787c659ff1"} Nov 02 23:28:15.213030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.212986 5672 scope.go:122] "RemoveContainer" containerID="29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a" Nov 02 23:28:15.219914 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.219887 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgqhp\" (UniqueName: \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-kube-api-access-cgqhp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.220045 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.219918 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.220045 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.219935 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.220045 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.219948 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8581a625-fd4b-47d0-9ac1-08f312407c1b-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:15.227200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.227181 5672 scope.go:122] "RemoveContainer" containerID="29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a" Nov 02 23:28:15.227543 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:28:15.227503 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a\": container with ID starting with 29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a not found: ID does not exist" containerID="29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a" Nov 02 23:28:15.227643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.227542 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a"} err="failed to get container status \"29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a\": rpc error: code = NotFound desc = could not find container \"29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a\": container with ID starting with 29e49f8cfeef6787d9c786a5b0b3cce2737cd0d548bf0a8090e7609b34ab429a not found: ID does not exist" Nov 02 23:28:15.236041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.236015 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:28:15.374345 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.374261 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws"] Nov 02 23:28:15.378219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.378185 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-cgqws"] Nov 02 23:28:15.540476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.540435 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp"] Nov 02 23:28:15.646492 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:15.646406 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95"] Nov 02 23:28:16.713415 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:16.713383 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" path="/var/lib/kubelet/pods/8581a625-fd4b-47d0-9ac1-08f312407c1b/volumes" Nov 02 23:28:18.716756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:18.716723 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr"] Nov 02 23:28:18.717168 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:18.716925 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" containerName="istio-proxy" containerID="cri-o://fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a" gracePeriod=30 Nov 02 23:28:20.541162 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.541086 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" podUID="c05b59e1-7bb2-493e-b598-edd970ce8c2c" containerName="payload-processing" containerID="cri-o://f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7" gracePeriod=25 Nov 02 23:28:20.647110 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.647058 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" podUID="abaf2bda-f79e-4510-9d60-a65cb7d709e1" containerName="payload-pre-processing" containerID="cri-o://1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab" gracePeriod=25 Nov 02 23:28:20.655136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.655037 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:28:20.655278 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.655206 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:28:20.804224 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.804185 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:28:20.888296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.888264 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05b59e1-7bb2-493e-b598-edd970ce8c2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05b59e1-7bb2-493e-b598-edd970ce8c2c-plugins-config-volume\") pod \"c05b59e1-7bb2-493e-b598-edd970ce8c2c\" (UID: \"c05b59e1-7bb2-493e-b598-edd970ce8c2c\") " Nov 02 23:28:20.888464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.888311 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05b59e1-7bb2-493e-b598-edd970ce8c2c-kube-api-access-hbjfr\" (UniqueName: \"kubernetes.io/projected/c05b59e1-7bb2-493e-b598-edd970ce8c2c-kube-api-access-hbjfr\") pod \"c05b59e1-7bb2-493e-b598-edd970ce8c2c\" (UID: \"c05b59e1-7bb2-493e-b598-edd970ce8c2c\") " Nov 02 23:28:20.888708 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.888681 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05b59e1-7bb2-493e-b598-edd970ce8c2c-plugins-config-volume" pod "c05b59e1-7bb2-493e-b598-edd970ce8c2c" (UID: "c05b59e1-7bb2-493e-b598-edd970ce8c2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:20.890315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.890284 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05b59e1-7bb2-493e-b598-edd970ce8c2c-kube-api-access-hbjfr" pod "c05b59e1-7bb2-493e-b598-edd970ce8c2c" (UID: "c05b59e1-7bb2-493e-b598-edd970ce8c2c"). InnerVolumeSpecName "kube-api-access-hbjfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:20.916083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.916061 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:28:20.989612 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.989575 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abaf2bda-f79e-4510-9d60-a65cb7d709e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abaf2bda-f79e-4510-9d60-a65cb7d709e1-plugins-config-volume\") pod \"abaf2bda-f79e-4510-9d60-a65cb7d709e1\" (UID: \"abaf2bda-f79e-4510-9d60-a65cb7d709e1\") " Nov 02 23:28:20.989739 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.989707 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abaf2bda-f79e-4510-9d60-a65cb7d709e1-kube-api-access-ffflq\" (UniqueName: \"kubernetes.io/projected/abaf2bda-f79e-4510-9d60-a65cb7d709e1-kube-api-access-ffflq\") pod \"abaf2bda-f79e-4510-9d60-a65cb7d709e1\" (UID: \"abaf2bda-f79e-4510-9d60-a65cb7d709e1\") " Nov 02 23:28:20.989931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.989909 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abaf2bda-f79e-4510-9d60-a65cb7d709e1-plugins-config-volume" pod "abaf2bda-f79e-4510-9d60-a65cb7d709e1" (UID: "abaf2bda-f79e-4510-9d60-a65cb7d709e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:20.990131 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.990115 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05b59e1-7bb2-493e-b598-edd970ce8c2c-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:20.990184 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.990133 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbjfr\" (UniqueName: \"kubernetes.io/projected/c05b59e1-7bb2-493e-b598-edd970ce8c2c-kube-api-access-hbjfr\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:20.990184 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.990142 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abaf2bda-f79e-4510-9d60-a65cb7d709e1-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:20.991816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:20.991795 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abaf2bda-f79e-4510-9d60-a65cb7d709e1-kube-api-access-ffflq" pod "abaf2bda-f79e-4510-9d60-a65cb7d709e1" (UID: "abaf2bda-f79e-4510-9d60-a65cb7d709e1"). InnerVolumeSpecName "kube-api-access-ffflq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:21.091550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.091475 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffflq\" (UniqueName: \"kubernetes.io/projected/abaf2bda-f79e-4510-9d60-a65cb7d709e1-kube-api-access-ffflq\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:21.192376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.192346 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:28:21.192537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.192423 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:28:21.257302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.257263 5672 generic.go:356] "Generic (PLEG): container finished" podID="abaf2bda-f79e-4510-9d60-a65cb7d709e1" containerID="1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab" exitCode=0 Nov 02 23:28:21.257455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.257319 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" Nov 02 23:28:21.257455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.257345 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" event={"ID":"abaf2bda-f79e-4510-9d60-a65cb7d709e1","Type":"ContainerDied","Data":"1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab"} Nov 02 23:28:21.257455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.257401 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95" event={"ID":"abaf2bda-f79e-4510-9d60-a65cb7d709e1","Type":"ContainerDied","Data":"e0fb572c115453ba84c2aa1b8e6e79da7aad079357c114902f6c5b8952666bc2"} Nov 02 23:28:21.257455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.257428 5672 scope.go:122] "RemoveContainer" containerID="1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab" Nov 02 23:28:21.259115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.259088 5672 generic.go:356] "Generic (PLEG): container finished" podID="c05b59e1-7bb2-493e-b598-edd970ce8c2c" containerID="f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7" exitCode=0 Nov 02 23:28:21.259255 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.259117 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" event={"ID":"c05b59e1-7bb2-493e-b598-edd970ce8c2c","Type":"ContainerDied","Data":"f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7"} Nov 02 23:28:21.259255 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.259144 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" Nov 02 23:28:21.259255 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.259186 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp" event={"ID":"c05b59e1-7bb2-493e-b598-edd970ce8c2c","Type":"ContainerDied","Data":"bb3cd6b79e2c6fe82d15c7710cfffcdd5530b10b62f92d756e4f384fee6dbc51"} Nov 02 23:28:21.275594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.275572 5672 scope.go:122] "RemoveContainer" containerID="1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab" Nov 02 23:28:21.275901 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:28:21.275873 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab\": container with ID starting with 1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab not found: ID does not exist" containerID="1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab" Nov 02 23:28:21.276011 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.275908 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab"} err="failed to get container status \"1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab\": rpc error: code = NotFound desc = could not find container \"1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab\": container with ID starting with 1eaf06ae88c5baa23533a6d544d0c464978b710c2302760807a26579fdab1eab not found: ID does not exist" Nov 02 23:28:21.276011 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.275930 5672 scope.go:122] "RemoveContainer" containerID="f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7" Nov 02 23:28:21.284562 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.284545 5672 scope.go:122] "RemoveContainer" containerID="f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7" Nov 02 23:28:21.284814 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:28:21.284795 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7\": container with ID starting with f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7 not found: ID does not exist" containerID="f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7" Nov 02 23:28:21.284863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.284822 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7"} err="failed to get container status \"f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7\": rpc error: code = NotFound desc = could not find container \"f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7\": container with ID starting with f9f274caf39bf0eb0a406d72982d81a00070d56d17d872e9c133e2d77993bbe7 not found: ID does not exist" Nov 02 23:28:21.509300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.509267 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95"] Nov 02 23:28:21.514038 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.514004 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-443fdf-54df594dcf-6dp95"] Nov 02 23:28:21.614805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.614780 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp"] Nov 02 23:28:21.617089 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:21.617064 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-443fdf-66d5885f6c-g4lkp"] Nov 02 23:28:22.713044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:22.713005 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abaf2bda-f79e-4510-9d60-a65cb7d709e1" path="/var/lib/kubelet/pods/abaf2bda-f79e-4510-9d60-a65cb7d709e1/volumes" Nov 02 23:28:22.713607 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:22.713584 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05b59e1-7bb2-493e-b598-edd970ce8c2c" path="/var/lib/kubelet/pods/c05b59e1-7bb2-493e-b598-edd970ce8c2c/volumes" Nov 02 23:28:23.983799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:23.983741 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:28:24.021309 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021277 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-kube-api-access-p55l2\" (UniqueName: \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-kube-api-access-p55l2\") pod \"b53e1f95-aa31-4384-a917-f1392d50907d\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " Nov 02 23:28:24.021458 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021324 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-istio-token\" (UniqueName: \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-istio-token\") pod \"b53e1f95-aa31-4384-a917-f1392d50907d\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " Nov 02 23:28:24.021458 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021394 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-credential-socket\") pod \"b53e1f95-aa31-4384-a917-f1392d50907d\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " Nov 02 23:28:24.021571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021502 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-certs\") pod \"b53e1f95-aa31-4384-a917-f1392d50907d\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " Nov 02 23:28:24.021571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021544 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-envoy\") pod \"b53e1f95-aa31-4384-a917-f1392d50907d\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " Nov 02 23:28:24.021671 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021612 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b53e1f95-aa31-4384-a917-f1392d50907d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53e1f95-aa31-4384-a917-f1392d50907d-istiod-ca-cert\") pod \"b53e1f95-aa31-4384-a917-f1392d50907d\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " Nov 02 23:28:24.021725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021670 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-socket\") pod \"b53e1f95-aa31-4384-a917-f1392d50907d\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " Nov 02 23:28:24.021725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021707 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-credential-socket" pod "b53e1f95-aa31-4384-a917-f1392d50907d" (UID: "b53e1f95-aa31-4384-a917-f1392d50907d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:24.021829 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021725 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b53e1f95-aa31-4384-a917-f1392d50907d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53e1f95-aa31-4384-a917-f1392d50907d-istio-podinfo\") pod \"b53e1f95-aa31-4384-a917-f1392d50907d\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " Nov 02 23:28:24.021829 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021728 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-certs" pod "b53e1f95-aa31-4384-a917-f1392d50907d" (UID: "b53e1f95-aa31-4384-a917-f1392d50907d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:24.021829 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021758 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-data\") pod \"b53e1f95-aa31-4384-a917-f1392d50907d\" (UID: \"b53e1f95-aa31-4384-a917-f1392d50907d\") " Nov 02 23:28:24.021965 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.021918 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-socket" pod "b53e1f95-aa31-4384-a917-f1392d50907d" (UID: "b53e1f95-aa31-4384-a917-f1392d50907d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:24.022126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.022027 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53e1f95-aa31-4384-a917-f1392d50907d-istiod-ca-cert" pod "b53e1f95-aa31-4384-a917-f1392d50907d" (UID: "b53e1f95-aa31-4384-a917-f1392d50907d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:24.022221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.022093 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-data" pod "b53e1f95-aa31-4384-a917-f1392d50907d" (UID: "b53e1f95-aa31-4384-a917-f1392d50907d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:24.022221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.022184 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:24.022221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.022201 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:24.022221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.022210 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:24.023978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.023951 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-envoy" pod "b53e1f95-aa31-4384-a917-f1392d50907d" (UID: "b53e1f95-aa31-4384-a917-f1392d50907d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:24.024082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.023985 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-istio-token" pod "b53e1f95-aa31-4384-a917-f1392d50907d" (UID: "b53e1f95-aa31-4384-a917-f1392d50907d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:24.024082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.024042 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-kube-api-access-p55l2" pod "b53e1f95-aa31-4384-a917-f1392d50907d" (UID: "b53e1f95-aa31-4384-a917-f1392d50907d"). InnerVolumeSpecName "kube-api-access-p55l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:24.024082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.024039 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b53e1f95-aa31-4384-a917-f1392d50907d-istio-podinfo" pod "b53e1f95-aa31-4384-a917-f1392d50907d" (UID: "b53e1f95-aa31-4384-a917-f1392d50907d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:28:24.123284 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.123253 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:24.123284 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.123280 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53e1f95-aa31-4384-a917-f1392d50907d-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:24.123284 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.123291 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53e1f95-aa31-4384-a917-f1392d50907d-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:24.123505 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.123301 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53e1f95-aa31-4384-a917-f1392d50907d-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:24.123505 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.123309 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p55l2\" (UniqueName: \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-kube-api-access-p55l2\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:24.123505 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.123318 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53e1f95-aa31-4384-a917-f1392d50907d-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:24.281939 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.281850 5672 generic.go:356] "Generic (PLEG): container finished" podID="b53e1f95-aa31-4384-a917-f1392d50907d" containerID="fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a" exitCode=0 Nov 02 23:28:24.281939 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.281916 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" Nov 02 23:28:24.282138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.281932 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" event={"ID":"b53e1f95-aa31-4384-a917-f1392d50907d","Type":"ContainerDied","Data":"fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a"} Nov 02 23:28:24.282138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.282017 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr" event={"ID":"b53e1f95-aa31-4384-a917-f1392d50907d","Type":"ContainerDied","Data":"38b321bf6c0b95a766362348aed03ab2be6337a2874deb298a2c2615af745d42"} Nov 02 23:28:24.282138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.282040 5672 scope.go:122] "RemoveContainer" containerID="fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a" Nov 02 23:28:24.293838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.293817 5672 scope.go:122] "RemoveContainer" containerID="fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a" Nov 02 23:28:24.294127 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:28:24.294106 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a\": container with ID starting with fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a not found: ID does not exist" containerID="fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a" Nov 02 23:28:24.294204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.294135 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a"} err="failed to get container status \"fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a\": rpc error: code = NotFound desc = could not find container \"fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a\": container with ID starting with fb96b50a3aa4fd6ecf56717b8979788118b3cdc53e2e0df92af1577752d3784a not found: ID does not exist" Nov 02 23:28:24.436847 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.436816 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr"] Nov 02 23:28:24.440977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.440948 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-7kxtr"] Nov 02 23:28:24.716524 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:24.716437 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" path="/var/lib/kubelet/pods/b53e1f95-aa31-4384-a917-f1392d50907d/volumes" Nov 02 23:28:26.020889 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:28:26.020846 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:28:28.057117 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:28.057082 5672 scope.go:122] "RemoveContainer" containerID="47dff56f8bfafc357320da889739c681c35de59727a9b6ffb58ff8893ac9cd58" Nov 02 23:28:41.686688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:41.686650 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:28:48.718373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:48.718315 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg"] Nov 02 23:28:48.718996 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:48.718502 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerName="istio-proxy" containerID="cri-o://9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee" gracePeriod=30 Nov 02 23:28:51.950680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:51.950644 5672 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:28:51.951060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:51.950701 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:28:53.976485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:53.976458 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:28:54.044435 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.044344 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-credential-socket\") pod \"d18cd3f1-549c-4fee-9cfd-f6844650c560\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " Nov 02 23:28:54.044435 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.044430 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d18cd3f1-549c-4fee-9cfd-f6844650c560-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18cd3f1-549c-4fee-9cfd-f6844650c560-istiod-ca-cert\") pod \"d18cd3f1-549c-4fee-9cfd-f6844650c560\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " Nov 02 23:28:54.044663 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.044509 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-token\" (UniqueName: \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-token\") pod \"d18cd3f1-549c-4fee-9cfd-f6844650c560\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " Nov 02 23:28:54.044734 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.044686 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-socket\") pod \"d18cd3f1-549c-4fee-9cfd-f6844650c560\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " Nov 02 23:28:54.044734 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.044691 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-credential-socket" pod "d18cd3f1-549c-4fee-9cfd-f6844650c560" (UID: "d18cd3f1-549c-4fee-9cfd-f6844650c560"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:54.044926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.044906 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-socket" pod "d18cd3f1-549c-4fee-9cfd-f6844650c560" (UID: "d18cd3f1-549c-4fee-9cfd-f6844650c560"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:54.045015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.044980 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-certs\") pod \"d18cd3f1-549c-4fee-9cfd-f6844650c560\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " Nov 02 23:28:54.045106 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.045046 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-kube-api-access-mtkz2\" (UniqueName: \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-kube-api-access-mtkz2\") pod \"d18cd3f1-549c-4fee-9cfd-f6844650c560\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " Nov 02 23:28:54.045106 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.045088 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-podinfo\") pod \"d18cd3f1-549c-4fee-9cfd-f6844650c560\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " Nov 02 23:28:54.045106 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.045073 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18cd3f1-549c-4fee-9cfd-f6844650c560-istiod-ca-cert" pod "d18cd3f1-549c-4fee-9cfd-f6844650c560" (UID: "d18cd3f1-549c-4fee-9cfd-f6844650c560"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:28:54.045402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.045120 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-data\") pod \"d18cd3f1-549c-4fee-9cfd-f6844650c560\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " Nov 02 23:28:54.045402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.045165 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-envoy\") pod \"d18cd3f1-549c-4fee-9cfd-f6844650c560\" (UID: \"d18cd3f1-549c-4fee-9cfd-f6844650c560\") " Nov 02 23:28:54.045402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.045248 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-certs" pod "d18cd3f1-549c-4fee-9cfd-f6844650c560" (UID: "d18cd3f1-549c-4fee-9cfd-f6844650c560"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:54.045650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.045583 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:54.045650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.045599 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18cd3f1-549c-4fee-9cfd-f6844650c560-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:54.045650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.045613 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:54.045650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.045625 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:54.045981 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.045957 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-data" pod "d18cd3f1-549c-4fee-9cfd-f6844650c560" (UID: "d18cd3f1-549c-4fee-9cfd-f6844650c560"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:54.047475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.047445 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-token" pod "d18cd3f1-549c-4fee-9cfd-f6844650c560" (UID: "d18cd3f1-549c-4fee-9cfd-f6844650c560"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:54.047746 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.047712 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-kube-api-access-mtkz2" pod "d18cd3f1-549c-4fee-9cfd-f6844650c560" (UID: "d18cd3f1-549c-4fee-9cfd-f6844650c560"). InnerVolumeSpecName "kube-api-access-mtkz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:28:54.047860 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.047841 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-podinfo" pod "d18cd3f1-549c-4fee-9cfd-f6844650c560" (UID: "d18cd3f1-549c-4fee-9cfd-f6844650c560"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:28:54.048231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.048210 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-envoy" pod "d18cd3f1-549c-4fee-9cfd-f6844650c560" (UID: "d18cd3f1-549c-4fee-9cfd-f6844650c560"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:28:54.146546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.146515 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtkz2\" (UniqueName: \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-kube-api-access-mtkz2\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:54.146546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.146544 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:54.146546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.146554 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:54.146869 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.146567 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:54.146869 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.146582 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18cd3f1-549c-4fee-9cfd-f6844650c560-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:28:54.480913 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.480877 5672 generic.go:356] "Generic (PLEG): container finished" podID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerID="9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee" exitCode=0 Nov 02 23:28:54.481082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.480926 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" Nov 02 23:28:54.481082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.480954 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" event={"ID":"d18cd3f1-549c-4fee-9cfd-f6844650c560","Type":"ContainerDied","Data":"9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee"} Nov 02 23:28:54.481082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.481007 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg" event={"ID":"d18cd3f1-549c-4fee-9cfd-f6844650c560","Type":"ContainerDied","Data":"7664ad3e88f9e761f3951eb28ff899f2d90502819893dfaf87eeb1cf2313f5df"} Nov 02 23:28:54.481082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.481029 5672 scope.go:122] "RemoveContainer" containerID="9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee" Nov 02 23:28:54.491893 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.491876 5672 scope.go:122] "RemoveContainer" containerID="9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee" Nov 02 23:28:54.492136 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:28:54.492119 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee\": container with ID starting with 9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee not found: ID does not exist" containerID="9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee" Nov 02 23:28:54.492205 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.492143 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee"} err="failed to get container status \"9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee\": rpc error: code = NotFound desc = could not find container \"9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee\": container with ID starting with 9427cd67f3e7c32aa83047ca7eaab9ce48fd1ef23b9bae11d916456fcc7360ee not found: ID does not exist" Nov 02 23:28:54.634575 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.634541 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg"] Nov 02 23:28:54.680730 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.680701 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-8nnrg"] Nov 02 23:28:54.713335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:28:54.713306 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" path="/var/lib/kubelet/pods/d18cd3f1-549c-4fee-9cfd-f6844650c560/volumes" Nov 02 23:28:56.020728 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:28:56.020687 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:29:18.718878 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:18.718836 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh"] Nov 02 23:29:18.719329 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:18.719057 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" containerID="cri-o://0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34" gracePeriod=30 Nov 02 23:29:21.311701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:21.311667 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h"] Nov 02 23:29:21.417424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:21.417389 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4"] Nov 02 23:29:23.987052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:23.986992 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:29:24.064559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.064525 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-token\" (UniqueName: \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-token\") pod \"80910ec3-8b61-4304-a410-d4bca2b27fe7\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " Nov 02 23:29:24.064745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.064612 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-data\") pod \"80910ec3-8b61-4304-a410-d4bca2b27fe7\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " Nov 02 23:29:24.064745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.064645 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-envoy\") pod \"80910ec3-8b61-4304-a410-d4bca2b27fe7\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " Nov 02 23:29:24.064745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.064681 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-podinfo\") pod \"80910ec3-8b61-4304-a410-d4bca2b27fe7\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " Nov 02 23:29:24.065054 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.065028 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-credential-socket\") pod \"80910ec3-8b61-4304-a410-d4bca2b27fe7\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " Nov 02 23:29:24.065202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.065061 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-data" pod "80910ec3-8b61-4304-a410-d4bca2b27fe7" (UID: "80910ec3-8b61-4304-a410-d4bca2b27fe7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:24.065302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.065275 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-credential-socket" pod "80910ec3-8b61-4304-a410-d4bca2b27fe7" (UID: "80910ec3-8b61-4304-a410-d4bca2b27fe7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:24.065406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.065358 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80910ec3-8b61-4304-a410-d4bca2b27fe7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80910ec3-8b61-4304-a410-d4bca2b27fe7-istiod-ca-cert\") pod \"80910ec3-8b61-4304-a410-d4bca2b27fe7\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " Nov 02 23:29:24.065476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.065425 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-socket\") pod \"80910ec3-8b61-4304-a410-d4bca2b27fe7\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " Nov 02 23:29:24.065577 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.065558 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-kube-api-access-fkb4p\" (UniqueName: \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-kube-api-access-fkb4p\") pod \"80910ec3-8b61-4304-a410-d4bca2b27fe7\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " Nov 02 23:29:24.065642 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.065596 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-certs\") pod \"80910ec3-8b61-4304-a410-d4bca2b27fe7\" (UID: \"80910ec3-8b61-4304-a410-d4bca2b27fe7\") " Nov 02 23:29:24.065712 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.065659 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-socket" pod "80910ec3-8b61-4304-a410-d4bca2b27fe7" (UID: "80910ec3-8b61-4304-a410-d4bca2b27fe7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:24.065773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.065733 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80910ec3-8b61-4304-a410-d4bca2b27fe7-istiod-ca-cert" pod "80910ec3-8b61-4304-a410-d4bca2b27fe7" (UID: "80910ec3-8b61-4304-a410-d4bca2b27fe7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:24.065981 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.065962 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-certs" pod "80910ec3-8b61-4304-a410-d4bca2b27fe7" (UID: "80910ec3-8b61-4304-a410-d4bca2b27fe7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:24.066084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.066000 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:24.066084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.066060 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:24.066084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.066075 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:24.066248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.066088 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80910ec3-8b61-4304-a410-d4bca2b27fe7-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:24.067171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.067126 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-token" pod "80910ec3-8b61-4304-a410-d4bca2b27fe7" (UID: "80910ec3-8b61-4304-a410-d4bca2b27fe7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:24.067369 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.067342 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-podinfo" pod "80910ec3-8b61-4304-a410-d4bca2b27fe7" (UID: "80910ec3-8b61-4304-a410-d4bca2b27fe7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:29:24.067788 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.067761 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-envoy" pod "80910ec3-8b61-4304-a410-d4bca2b27fe7" (UID: "80910ec3-8b61-4304-a410-d4bca2b27fe7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:29:24.067939 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.067915 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-kube-api-access-fkb4p" pod "80910ec3-8b61-4304-a410-d4bca2b27fe7" (UID: "80910ec3-8b61-4304-a410-d4bca2b27fe7"). InnerVolumeSpecName "kube-api-access-fkb4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:24.167457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.167422 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkb4p\" (UniqueName: \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-kube-api-access-fkb4p\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:24.167457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.167460 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:24.167670 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.167475 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:24.167670 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.167488 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:24.167670 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.167502 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80910ec3-8b61-4304-a410-d4bca2b27fe7-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:24.677846 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.677806 5672 generic.go:356] "Generic (PLEG): container finished" podID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerID="0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34" exitCode=0 Nov 02 23:29:24.678014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.677871 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" Nov 02 23:29:24.678014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.677888 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" event={"ID":"80910ec3-8b61-4304-a410-d4bca2b27fe7","Type":"ContainerDied","Data":"0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34"} Nov 02 23:29:24.678014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.677965 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh" event={"ID":"80910ec3-8b61-4304-a410-d4bca2b27fe7","Type":"ContainerDied","Data":"36d7ab01bde8cf72d7f058a2af0480bdc165681d7774d4357e3e992a17feef36"} Nov 02 23:29:24.678014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.677989 5672 scope.go:122] "RemoveContainer" containerID="0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34" Nov 02 23:29:24.688468 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.688452 5672 scope.go:122] "RemoveContainer" containerID="0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34" Nov 02 23:29:24.688717 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:29:24.688700 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34\": container with ID starting with 0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34 not found: ID does not exist" containerID="0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34" Nov 02 23:29:24.688756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.688726 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34"} err="failed to get container status \"0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34\": rpc error: code = NotFound desc = could not find container \"0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34\": container with ID starting with 0c805592e32ca5a793660df60441d5dbb69ffb4d84d018983a23d489121e8e34 not found: ID does not exist" Nov 02 23:29:24.879235 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.879200 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh"] Nov 02 23:29:24.925485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:24.925447 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-xtrqh"] Nov 02 23:29:25.557328 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:25.557292 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:29:26.020831 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:29:26.020797 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:29:26.312281 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.312189 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" podUID="c13ce135-1fe4-494a-b161-a74545a17d44" containerName="payload-processing" containerID="cri-o://3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae" gracePeriod=25 Nov 02 23:29:26.417731 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.417667 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" podUID="2620c879-d7a2-46cf-a94e-6624cc4a291a" containerName="payload-pre-processing" containerID="cri-o://156134ae71928789b2a439221628db6e4358744a692c6bf02e8a5ef931227382" gracePeriod=25 Nov 02 23:29:26.600774 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.600747 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:29:26.692625 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.692594 5672 generic.go:356] "Generic (PLEG): container finished" podID="2620c879-d7a2-46cf-a94e-6624cc4a291a" containerID="156134ae71928789b2a439221628db6e4358744a692c6bf02e8a5ef931227382" exitCode=0 Nov 02 23:29:26.692779 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.692694 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" event={"ID":"2620c879-d7a2-46cf-a94e-6624cc4a291a","Type":"ContainerDied","Data":"156134ae71928789b2a439221628db6e4358744a692c6bf02e8a5ef931227382"} Nov 02 23:29:26.694051 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.694026 5672 generic.go:356] "Generic (PLEG): container finished" podID="c13ce135-1fe4-494a-b161-a74545a17d44" containerID="3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae" exitCode=0 Nov 02 23:29:26.694161 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.694110 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" event={"ID":"c13ce135-1fe4-494a-b161-a74545a17d44","Type":"ContainerDied","Data":"3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae"} Nov 02 23:29:26.694161 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.694139 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" event={"ID":"c13ce135-1fe4-494a-b161-a74545a17d44","Type":"ContainerDied","Data":"b702768a6c33dcbe04b91bfa8f0ce3531e5546dc6d58ec18032bd4b00ede0528"} Nov 02 23:29:26.694239 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.694178 5672 scope.go:122] "RemoveContainer" containerID="3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae" Nov 02 23:29:26.694295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.694270 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h" Nov 02 23:29:26.704280 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.704253 5672 scope.go:122] "RemoveContainer" containerID="3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae" Nov 02 23:29:26.704553 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:29:26.704531 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae\": container with ID starting with 3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae not found: ID does not exist" containerID="3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae" Nov 02 23:29:26.704629 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.704560 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae"} err="failed to get container status \"3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae\": rpc error: code = NotFound desc = could not find container \"3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae\": container with ID starting with 3e6f734cef8dbc4a24c4c792d2bbbd5e3787114f85973f73c718c41109d5adae not found: ID does not exist" Nov 02 23:29:26.713881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.713858 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" path="/var/lib/kubelet/pods/80910ec3-8b61-4304-a410-d4bca2b27fe7/volumes" Nov 02 23:29:26.715653 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.715633 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13ce135-1fe4-494a-b161-a74545a17d44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13ce135-1fe4-494a-b161-a74545a17d44-plugins-config-volume\") pod \"c13ce135-1fe4-494a-b161-a74545a17d44\" (UID: \"c13ce135-1fe4-494a-b161-a74545a17d44\") " Nov 02 23:29:26.715750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.715732 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13ce135-1fe4-494a-b161-a74545a17d44-kube-api-access-c2b8r\" (UniqueName: \"kubernetes.io/projected/c13ce135-1fe4-494a-b161-a74545a17d44-kube-api-access-c2b8r\") pod \"c13ce135-1fe4-494a-b161-a74545a17d44\" (UID: \"c13ce135-1fe4-494a-b161-a74545a17d44\") " Nov 02 23:29:26.715948 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.715927 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13ce135-1fe4-494a-b161-a74545a17d44-plugins-config-volume" pod "c13ce135-1fe4-494a-b161-a74545a17d44" (UID: "c13ce135-1fe4-494a-b161-a74545a17d44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:26.716093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.716077 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13ce135-1fe4-494a-b161-a74545a17d44-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:26.717780 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.717753 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13ce135-1fe4-494a-b161-a74545a17d44-kube-api-access-c2b8r" pod "c13ce135-1fe4-494a-b161-a74545a17d44" (UID: "c13ce135-1fe4-494a-b161-a74545a17d44"). InnerVolumeSpecName "kube-api-access-c2b8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:26.766861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.766833 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:29:26.816811 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.816729 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2b8r\" (UniqueName: \"kubernetes.io/projected/c13ce135-1fe4-494a-b161-a74545a17d44-kube-api-access-c2b8r\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:26.917287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.917254 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2620c879-d7a2-46cf-a94e-6624cc4a291a-kube-api-access-2d8c5\" (UniqueName: \"kubernetes.io/projected/2620c879-d7a2-46cf-a94e-6624cc4a291a-kube-api-access-2d8c5\") pod \"2620c879-d7a2-46cf-a94e-6624cc4a291a\" (UID: \"2620c879-d7a2-46cf-a94e-6624cc4a291a\") " Nov 02 23:29:26.917457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.917368 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2620c879-d7a2-46cf-a94e-6624cc4a291a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2620c879-d7a2-46cf-a94e-6624cc4a291a-plugins-config-volume\") pod \"2620c879-d7a2-46cf-a94e-6624cc4a291a\" (UID: \"2620c879-d7a2-46cf-a94e-6624cc4a291a\") " Nov 02 23:29:26.917735 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.917710 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2620c879-d7a2-46cf-a94e-6624cc4a291a-plugins-config-volume" pod "2620c879-d7a2-46cf-a94e-6624cc4a291a" (UID: "2620c879-d7a2-46cf-a94e-6624cc4a291a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:29:26.919303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:26.919282 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2620c879-d7a2-46cf-a94e-6624cc4a291a-kube-api-access-2d8c5" pod "2620c879-d7a2-46cf-a94e-6624cc4a291a" (UID: "2620c879-d7a2-46cf-a94e-6624cc4a291a"). InnerVolumeSpecName "kube-api-access-2d8c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:29:27.018595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:27.018566 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2620c879-d7a2-46cf-a94e-6624cc4a291a-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:27.018595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:27.018593 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d8c5\" (UniqueName: \"kubernetes.io/projected/2620c879-d7a2-46cf-a94e-6624cc4a291a-kube-api-access-2d8c5\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:29:27.108132 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:27.108044 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h"] Nov 02 23:29:27.112102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:27.112076 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5ff336-6b9c5dfbb5-nw68h"] Nov 02 23:29:27.702315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:27.702276 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" Nov 02 23:29:27.702315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:27.702295 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4" event={"ID":"2620c879-d7a2-46cf-a94e-6624cc4a291a","Type":"ContainerDied","Data":"f87c5952738dbd1de510ecfdfb057eb460d43fcbfea4797058a5b1e4d593c576"} Nov 02 23:29:27.702769 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:27.702402 5672 scope.go:122] "RemoveContainer" containerID="156134ae71928789b2a439221628db6e4358744a692c6bf02e8a5ef931227382" Nov 02 23:29:27.858172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:27.858080 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4"] Nov 02 23:29:27.903212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:27.903180 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ff336-75dc89f96c-jcqm4"] Nov 02 23:29:28.712376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:28.712340 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2620c879-d7a2-46cf-a94e-6624cc4a291a" path="/var/lib/kubelet/pods/2620c879-d7a2-46cf-a94e-6624cc4a291a/volumes" Nov 02 23:29:28.712777 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:28.712762 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13ce135-1fe4-494a-b161-a74545a17d44" path="/var/lib/kubelet/pods/c13ce135-1fe4-494a-b161-a74545a17d44/volumes" Nov 02 23:29:32.004406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.004366 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz"] Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.004960 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2620c879-d7a2-46cf-a94e-6624cc4a291a" containerName="payload-pre-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.004976 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2620c879-d7a2-46cf-a94e-6624cc4a291a" containerName="payload-pre-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.004989 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerName="istio-proxy" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.004994 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerName="istio-proxy" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005002 5672 cpu_manager.go:414] "containerMap: removing container" podUID="abaf2bda-f79e-4510-9d60-a65cb7d709e1" containerName="payload-pre-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005006 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abaf2bda-f79e-4510-9d60-a65cb7d709e1" containerName="payload-pre-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005018 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerName="istio-proxy" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005023 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerName="istio-proxy" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005033 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c05b59e1-7bb2-493e-b598-edd970ce8c2c" containerName="payload-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005038 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05b59e1-7bb2-493e-b598-edd970ce8c2c" containerName="payload-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005050 5672 cpu_manager.go:414] "containerMap: removing container" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" containerName="istio-proxy" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005055 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" containerName="istio-proxy" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005064 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c13ce135-1fe4-494a-b161-a74545a17d44" containerName="payload-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005069 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13ce135-1fe4-494a-b161-a74545a17d44" containerName="payload-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005075 5672 cpu_manager.go:414] "containerMap: removing container" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005079 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005144 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2620c879-d7a2-46cf-a94e-6624cc4a291a" containerName="payload-pre-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005165 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18cd3f1-549c-4fee-9cfd-f6844650c560" containerName="istio-proxy" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005173 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="abaf2bda-f79e-4510-9d60-a65cb7d709e1" containerName="payload-pre-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005181 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8581a625-fd4b-47d0-9ac1-08f312407c1b" containerName="istio-proxy" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005189 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05b59e1-7bb2-493e-b598-edd970ce8c2c" containerName="payload-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005195 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="b53e1f95-aa31-4384-a917-f1392d50907d" containerName="istio-proxy" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005201 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13ce135-1fe4-494a-b161-a74545a17d44" containerName="payload-processing" Nov 02 23:29:32.006958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.005206 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="80910ec3-8b61-4304-a410-d4bca2b27fe7" containerName="istio-proxy" Nov 02 23:29:32.008387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.008370 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.083301 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.083269 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz"] Nov 02 23:29:32.103511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.103485 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5a3909-openshift-default-dockercfg-jsjst\"" Nov 02 23:29:32.195244 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.195188 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-data\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.195464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.195305 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-certs\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.195464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.195336 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-token\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.195464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.195383 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-podinfo\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.195464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.195433 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-socket\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.195464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.195459 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k7zx\" (UniqueName: \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-kube-api-access-2k7zx\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.195745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.195544 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-credential-socket\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.195745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.195576 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istiod-ca-cert\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.195745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.195695 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-envoy\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.296546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.296455 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-envoy\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.296837 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.296799 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-data\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.296966 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.296897 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-certs\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.296966 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.296927 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-token\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.297081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.296987 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-podinfo\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.297081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.297054 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-socket\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.297218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.297084 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k7zx\" (UniqueName: \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-kube-api-access-2k7zx\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.297218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.297200 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-credential-socket\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.297324 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.297233 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-data\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.297556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.297528 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-certs\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.297941 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.297909 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istiod-ca-cert\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.297941 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.297238 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istiod-ca-cert\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.298443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.298423 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-socket\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.298610 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.298587 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-credential-socket\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.301060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.301039 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-podinfo\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.301272 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.301256 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-envoy\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.351260 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.351228 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k7zx\" (UniqueName: \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-kube-api-access-2k7zx\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.351477 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.351273 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-token\") pod \"e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:32.629359 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:32.629255 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:33.081370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:33.078090 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz"] Nov 02 23:29:33.086223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:33.086184 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:29:33.086343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:33.086259 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:29:33.086343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:33.086294 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:29:33.741766 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:33.741723 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" event={"ID":"589ff327-0656-4cc4-ae2f-0c5c2c4d0059","Type":"ContainerStarted","Data":"8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6"} Nov 02 23:29:33.741943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:33.741791 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" event={"ID":"589ff327-0656-4cc4-ae2f-0c5c2c4d0059","Type":"ContainerStarted","Data":"83ba1cfe8750db6e488595c6af79c7ad441b550620bfd0957c442eac4884410b"} Nov 02 23:29:33.853060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:33.853002 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" podStartSLOduration=2.852985303 podStartE2EDuration="2.852985303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:33.85080715 +0000 UTC m=+2167.258118982" watchObservedRunningTime="2026-11-02 23:29:33.852985303 +0000 UTC m=+2167.260297130" Nov 02 23:29:34.629843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:34.629805 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:34.636284 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:34.636255 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:34.748541 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:34.748494 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:34.749544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:34.749524 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:29:35.756880 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:35.756798 5672 generic.go:356] "Generic (PLEG): container finished" podID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerID="f12c43a7c27fb401fa0cfc1a62126cc5c8dc8365281ccdca6bf71f6c831c95c5" exitCode=0 Nov 02 23:29:35.756880 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:35.756870 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" event={"ID":"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f","Type":"ContainerDied","Data":"f12c43a7c27fb401fa0cfc1a62126cc5c8dc8365281ccdca6bf71f6c831c95c5"} Nov 02 23:29:35.757504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:35.757486 5672 scope.go:122] "RemoveContainer" containerID="f12c43a7c27fb401fa0cfc1a62126cc5c8dc8365281ccdca6bf71f6c831c95c5" Nov 02 23:29:35.758943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:35.758915 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:29:35.759050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:35.758982 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:29:35.759050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:35.759014 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:29:36.770030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:36.769973 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" event={"ID":"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f","Type":"ContainerStarted","Data":"2754069ce4878620abcf94b60cbd83d89c43207d85be35d7024f4eb2261671b6"} Nov 02 23:29:37.445171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:37.444464 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:29:37.445171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:37.444580 5672 patch_prober.go:29] interesting pod/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 02 23:29:37.445171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:37.444624 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 02 23:29:38.453174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:38.448436 5672 patch_prober.go:29] interesting pod/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 02 23:29:38.453174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:38.448505 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 02 23:29:39.444175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:39.440700 5672 patch_prober.go:29] interesting pod/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 02 23:29:39.444175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:39.440759 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 02 23:29:40.447171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:40.446547 5672 patch_prober.go:29] interesting pod/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 02 23:29:40.447171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:40.446610 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 02 23:29:41.448535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.448499 5672 patch_prober.go:29] interesting pod/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 02 23:29:41.449122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.449092 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 02 23:29:41.593556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.593524 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4"] Nov 02 23:29:41.597432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.597410 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:29:41.609908 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.609879 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt672\" (UniqueName: \"kubernetes.io/projected/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-kube-api-access-rt672\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4\" (UID: \"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:29:41.610019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.609986 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4\" (UID: \"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:29:41.651871 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.651828 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4"] Nov 02 23:29:41.698207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.696593 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5a3909-dockercfg-drzx5\"" Nov 02 23:29:41.698207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.696660 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5a3909\"" Nov 02 23:29:41.712070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.711070 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt672\" (UniqueName: \"kubernetes.io/projected/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-kube-api-access-rt672\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4\" (UID: \"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:29:41.712070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.711169 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4\" (UID: \"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:29:41.712534 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.712504 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4\" (UID: \"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:29:41.752289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.752253 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g"] Nov 02 23:29:41.756220 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.756199 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:29:41.766855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.765401 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt672\" (UniqueName: \"kubernetes.io/projected/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-kube-api-access-rt672\") pod \"payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4\" (UID: \"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:29:41.810170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.809269 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g"] Nov 02 23:29:41.828199 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.814043 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rlh\" (UniqueName: \"kubernetes.io/projected/eec49111-55b2-4725-9582-44670bdc2f43-kube-api-access-p5rlh\") pod \"payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g\" (UID: \"eec49111-55b2-4725-9582-44670bdc2f43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:29:41.828199 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.814185 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec49111-55b2-4725-9582-44670bdc2f43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g\" (UID: \"eec49111-55b2-4725-9582-44670bdc2f43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:29:41.912035 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.911998 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:29:41.915857 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.915550 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec49111-55b2-4725-9582-44670bdc2f43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g\" (UID: \"eec49111-55b2-4725-9582-44670bdc2f43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:29:41.915857 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.915687 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rlh\" (UniqueName: \"kubernetes.io/projected/eec49111-55b2-4725-9582-44670bdc2f43-kube-api-access-p5rlh\") pod \"payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g\" (UID: \"eec49111-55b2-4725-9582-44670bdc2f43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:29:41.916429 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.916405 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec49111-55b2-4725-9582-44670bdc2f43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g\" (UID: \"eec49111-55b2-4725-9582-44670bdc2f43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:29:41.972410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:41.971126 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rlh\" (UniqueName: \"kubernetes.io/projected/eec49111-55b2-4725-9582-44670bdc2f43-kube-api-access-p5rlh\") pod \"payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g\" (UID: \"eec49111-55b2-4725-9582-44670bdc2f43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:29:42.082466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.082432 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:29:42.294653 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.294625 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4"] Nov 02 23:29:42.296521 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:29:42.296450 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424b53ac_a9b0_4f2c_ae87_3a34fcb001fc.slice/crio-1860bb11aeed6cf840f2fafe87c8490cad14d17c09b9bcbbc102947c4d1f27c3 WatchSource:0}: Error finding container 1860bb11aeed6cf840f2fafe87c8490cad14d17c09b9bcbbc102947c4d1f27c3: Status 404 returned error can't find the container with id 1860bb11aeed6cf840f2fafe87c8490cad14d17c09b9bcbbc102947c4d1f27c3 Nov 02 23:29:42.444559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.444529 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:29:42.444925 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.444897 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:29:42.445540 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.445521 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:29:42.453717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.449312 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g"] Nov 02 23:29:42.453717 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:29:42.453638 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec49111_55b2_4725_9582_44670bdc2f43.slice/crio-243babe8dfafd9e12aec695f8d30dfdac588fc5d0bb150b83ccd3f47b9566890 WatchSource:0}: Error finding container 243babe8dfafd9e12aec695f8d30dfdac588fc5d0bb150b83ccd3f47b9566890: Status 404 returned error can't find the container with id 243babe8dfafd9e12aec695f8d30dfdac588fc5d0bb150b83ccd3f47b9566890 Nov 02 23:29:42.809983 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.809950 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" event={"ID":"eec49111-55b2-4725-9582-44670bdc2f43","Type":"ContainerStarted","Data":"8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451"} Nov 02 23:29:42.810180 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.810005 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" event={"ID":"eec49111-55b2-4725-9582-44670bdc2f43","Type":"ContainerStarted","Data":"243babe8dfafd9e12aec695f8d30dfdac588fc5d0bb150b83ccd3f47b9566890"} Nov 02 23:29:42.810180 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.810031 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:29:42.811602 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.811574 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" event={"ID":"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc","Type":"ContainerStarted","Data":"f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6"} Nov 02 23:29:42.811714 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.811617 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" event={"ID":"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc","Type":"ContainerStarted","Data":"1860bb11aeed6cf840f2fafe87c8490cad14d17c09b9bcbbc102947c4d1f27c3"} Nov 02 23:29:42.811764 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.811751 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:29:42.918980 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:42.918928 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" podStartSLOduration=1.9189151949999999 podStartE2EDuration="1.918915195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:42.918505999 +0000 UTC m=+2176.325817834" watchObservedRunningTime="2026-11-02 23:29:42.918915195 +0000 UTC m=+2176.326227025" Nov 02 23:29:43.030943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:43.030889 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" podStartSLOduration=2.030876397 podStartE2EDuration="2.030876397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:43.029818367 +0000 UTC m=+2176.437130195" watchObservedRunningTime="2026-11-02 23:29:43.030876397 +0000 UTC m=+2176.438188226" Nov 02 23:29:46.396619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.396584 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l"] Nov 02 23:29:46.401030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.401004 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:29:46.474170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.467399 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l"] Nov 02 23:29:46.474170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.469071 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baed7942-8752-4757-8a2b-9b3993a91617-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l\" (UID: \"baed7942-8752-4757-8a2b-9b3993a91617\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:29:46.474170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.469107 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvscj\" (UniqueName: \"kubernetes.io/projected/baed7942-8752-4757-8a2b-9b3993a91617-kube-api-access-fvscj\") pod \"payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l\" (UID: \"baed7942-8752-4757-8a2b-9b3993a91617\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:29:46.503896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.503863 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-fb5fa2\"" Nov 02 23:29:46.503896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.503874 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-fb5fa2-dockercfg-4bmhn\"" Nov 02 23:29:46.553000 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.552970 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d"] Nov 02 23:29:46.556873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.556850 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:29:46.570922 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.570899 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2wg9\" (UniqueName: \"kubernetes.io/projected/523f9203-2495-4872-9f45-ab996220c4c7-kube-api-access-t2wg9\") pod \"payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d\" (UID: \"523f9203-2495-4872-9f45-ab996220c4c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:29:46.571069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.570963 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/523f9203-2495-4872-9f45-ab996220c4c7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d\" (UID: \"523f9203-2495-4872-9f45-ab996220c4c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:29:46.571166 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.571124 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baed7942-8752-4757-8a2b-9b3993a91617-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l\" (UID: \"baed7942-8752-4757-8a2b-9b3993a91617\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:29:46.571228 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.571179 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvscj\" (UniqueName: \"kubernetes.io/projected/baed7942-8752-4757-8a2b-9b3993a91617-kube-api-access-fvscj\") pod \"payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l\" (UID: \"baed7942-8752-4757-8a2b-9b3993a91617\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:29:46.571974 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.571956 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baed7942-8752-4757-8a2b-9b3993a91617-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l\" (UID: \"baed7942-8752-4757-8a2b-9b3993a91617\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:29:46.604504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.604475 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d"] Nov 02 23:29:46.624320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.624290 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvscj\" (UniqueName: \"kubernetes.io/projected/baed7942-8752-4757-8a2b-9b3993a91617-kube-api-access-fvscj\") pod \"payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l\" (UID: \"baed7942-8752-4757-8a2b-9b3993a91617\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:29:46.672682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.672608 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2wg9\" (UniqueName: \"kubernetes.io/projected/523f9203-2495-4872-9f45-ab996220c4c7-kube-api-access-t2wg9\") pod \"payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d\" (UID: \"523f9203-2495-4872-9f45-ab996220c4c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:29:46.672682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.672673 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/523f9203-2495-4872-9f45-ab996220c4c7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d\" (UID: \"523f9203-2495-4872-9f45-ab996220c4c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:29:46.673437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.673418 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/523f9203-2495-4872-9f45-ab996220c4c7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d\" (UID: \"523f9203-2495-4872-9f45-ab996220c4c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:29:46.714835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.714809 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:29:46.725737 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.725681 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2wg9\" (UniqueName: \"kubernetes.io/projected/523f9203-2495-4872-9f45-ab996220c4c7-kube-api-access-t2wg9\") pod \"payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d\" (UID: \"523f9203-2495-4872-9f45-ab996220c4c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:29:46.870729 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:46.870703 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:29:47.079863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:47.079828 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l"] Nov 02 23:29:47.081621 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:29:47.081595 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaed7942_8752_4757_8a2b_9b3993a91617.slice/crio-d9966b1a73c592312e68ac390ff50c61e307ebe8961516f62cb02a83c502d6f0 WatchSource:0}: Error finding container d9966b1a73c592312e68ac390ff50c61e307ebe8961516f62cb02a83c502d6f0: Status 404 returned error can't find the container with id d9966b1a73c592312e68ac390ff50c61e307ebe8961516f62cb02a83c502d6f0 Nov 02 23:29:47.238023 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:47.237995 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d"] Nov 02 23:29:47.238912 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:29:47.238863 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523f9203_2495_4872_9f45_ab996220c4c7.slice/crio-7deab8819e60a867225c198392c94960b0909b701ba7268aa4f1e9cf496f76c0 WatchSource:0}: Error finding container 7deab8819e60a867225c198392c94960b0909b701ba7268aa4f1e9cf496f76c0: Status 404 returned error can't find the container with id 7deab8819e60a867225c198392c94960b0909b701ba7268aa4f1e9cf496f76c0 Nov 02 23:29:47.842736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:47.842681 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" event={"ID":"baed7942-8752-4757-8a2b-9b3993a91617","Type":"ContainerStarted","Data":"b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a"} Nov 02 23:29:47.843261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:47.842787 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" event={"ID":"baed7942-8752-4757-8a2b-9b3993a91617","Type":"ContainerStarted","Data":"d9966b1a73c592312e68ac390ff50c61e307ebe8961516f62cb02a83c502d6f0"} Nov 02 23:29:47.843261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:47.842819 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:29:47.844456 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:47.844422 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" event={"ID":"523f9203-2495-4872-9f45-ab996220c4c7","Type":"ContainerStarted","Data":"c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa"} Nov 02 23:29:47.844599 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:47.844470 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" event={"ID":"523f9203-2495-4872-9f45-ab996220c4c7","Type":"ContainerStarted","Data":"7deab8819e60a867225c198392c94960b0909b701ba7268aa4f1e9cf496f76c0"} Nov 02 23:29:47.844599 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:47.844508 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:29:47.949772 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:47.949713 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" podStartSLOduration=1.9496978839999999 podStartE2EDuration="1.949697884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:47.949635417 +0000 UTC m=+2181.356947246" watchObservedRunningTime="2026-11-02 23:29:47.949697884 +0000 UTC m=+2181.357009715" Nov 02 23:29:48.058213 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:48.058161 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" podStartSLOduration=2.05813537 podStartE2EDuration="2.05813537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:48.054508141 +0000 UTC m=+2181.461819972" watchObservedRunningTime="2026-11-02 23:29:48.05813537 +0000 UTC m=+2181.465447237" Nov 02 23:29:53.284984 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:53.284940 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:29:53.821024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:53.820994 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:29:53.821227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:53.821075 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:29:56.021175 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:29:56.021119 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:29:58.855627 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:58.855594 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:29:58.856083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:29:58.855678 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:30:00.159027 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.158988 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x"] Nov 02 23:30:00.163837 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.163816 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" Nov 02 23:30:00.164748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.164723 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb"] Nov 02 23:30:00.168663 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.168642 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d"] Nov 02 23:30:00.168793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.168743 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" Nov 02 23:30:00.172541 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.172497 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" Nov 02 23:30:00.204097 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.204075 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh"] Nov 02 23:30:00.207581 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.207566 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t"] Nov 02 23:30:00.207670 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.207656 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" Nov 02 23:30:00.211052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.211031 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" Nov 02 23:30:00.211751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.211734 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g"] Nov 02 23:30:00.221200 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.221179 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n"] Nov 02 23:30:00.222483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.222458 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" Nov 02 23:30:00.222909 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.222883 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plqhw\" (UniqueName: \"kubernetes.io/projected/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-kube-api-access-plqhw\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb\" (UID: \"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" Nov 02 23:30:00.222982 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.222950 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfw92\" (UniqueName: \"kubernetes.io/projected/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-kube-api-access-tfw92\") pod \"maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x\" (UID: \"64237b78-b4e0-42c6-b9e6-3f75e7710ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" Nov 02 23:30:00.223198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.223137 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb\" (UID: \"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" Nov 02 23:30:00.223281 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.223262 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x\" (UID: \"64237b78-b4e0-42c6-b9e6-3f75e7710ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" Nov 02 23:30:00.226906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.226887 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78"] Nov 02 23:30:00.227356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.227336 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" Nov 02 23:30:00.232598 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.232578 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" Nov 02 23:30:00.248279 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.248261 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:30:00.248425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.248405 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:30:00.248499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.248425 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:30:00.252805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.252781 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d"] Nov 02 23:30:00.257370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.257349 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh"] Nov 02 23:30:00.259425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.259408 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x"] Nov 02 23:30:00.261437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.261418 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t"] Nov 02 23:30:00.309248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.309219 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n"] Nov 02 23:30:00.311814 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.311787 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78"] Nov 02 23:30:00.324870 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.324839 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plqhw\" (UniqueName: \"kubernetes.io/projected/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-kube-api-access-plqhw\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb\" (UID: \"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" Nov 02 23:30:00.325010 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.324888 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7025314-94d4-4994-a385-eae79b919e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78\" (UID: \"e7025314-94d4-4994-a385-eae79b919e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" Nov 02 23:30:00.325010 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.324918 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ms57\" (UniqueName: \"kubernetes.io/projected/e7025314-94d4-4994-a385-eae79b919e64-kube-api-access-2ms57\") pod \"maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78\" (UID: \"e7025314-94d4-4994-a385-eae79b919e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" Nov 02 23:30:00.325010 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.324947 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2da10-0859-46c8-a214-e90ce9f44840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n\" (UID: \"8fc2da10-0859-46c8-a214-e90ce9f44840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" Nov 02 23:30:00.325215 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325108 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nghv\" (UniqueName: \"kubernetes.io/projected/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-kube-api-access-9nghv\") pod \"maas-api-key-cleanup-29894370-jmk5t\" (UID: \"c67b7984-ede5-4e8f-b266-34cd84c7f2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" Nov 02 23:30:00.325215 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325178 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfw92\" (UniqueName: \"kubernetes.io/projected/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-kube-api-access-tfw92\") pod \"maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x\" (UID: \"64237b78-b4e0-42c6-b9e6-3f75e7710ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" Nov 02 23:30:00.325336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325230 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a900285e-ba2c-4da2-906b-b8c8f586441c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g\" (UID: \"a900285e-ba2c-4da2-906b-b8c8f586441c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" Nov 02 23:30:00.325394 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325358 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p77f\" (UniqueName: \"kubernetes.io/projected/a900285e-ba2c-4da2-906b-b8c8f586441c-kube-api-access-2p77f\") pod \"maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g\" (UID: \"a900285e-ba2c-4da2-906b-b8c8f586441c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" Nov 02 23:30:00.325449 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325393 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sxjk\" (UniqueName: \"kubernetes.io/projected/8fc2da10-0859-46c8-a214-e90ce9f44840-kube-api-access-9sxjk\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n\" (UID: \"8fc2da10-0859-46c8-a214-e90ce9f44840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" Nov 02 23:30:00.325502 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325444 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d\" (UID: \"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" Nov 02 23:30:00.325558 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325541 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q848\" (UniqueName: \"kubernetes.io/projected/88fc9bc8-8e69-4a72-887e-c3612ce0055f-kube-api-access-2q848\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh\" (UID: \"88fc9bc8-8e69-4a72-887e-c3612ce0055f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" Nov 02 23:30:00.325610 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325590 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fc9bc8-8e69-4a72-887e-c3612ce0055f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh\" (UID: \"88fc9bc8-8e69-4a72-887e-c3612ce0055f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" Nov 02 23:30:00.325650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325636 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894370-jmk5t\" (UID: \"c67b7984-ede5-4e8f-b266-34cd84c7f2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" Nov 02 23:30:00.325717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325703 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb\" (UID: \"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" Nov 02 23:30:00.325751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325737 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2xxw\" (UniqueName: \"kubernetes.io/projected/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-kube-api-access-j2xxw\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d\" (UID: \"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" Nov 02 23:30:00.325818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.325805 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x\" (UID: \"64237b78-b4e0-42c6-b9e6-3f75e7710ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" Nov 02 23:30:00.326695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.326675 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x\" (UID: \"64237b78-b4e0-42c6-b9e6-3f75e7710ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" Nov 02 23:30:00.327323 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.327304 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb\" (UID: \"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" Nov 02 23:30:00.353799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.353764 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g"] Nov 02 23:30:00.358131 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.357761 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb"] Nov 02 23:30:00.388760 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.388726 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfw92\" (UniqueName: \"kubernetes.io/projected/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-kube-api-access-tfw92\") pod \"maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x\" (UID: \"64237b78-b4e0-42c6-b9e6-3f75e7710ff1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" Nov 02 23:30:00.389788 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.389763 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plqhw\" (UniqueName: \"kubernetes.io/projected/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-kube-api-access-plqhw\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb\" (UID: \"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" Nov 02 23:30:00.426696 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.426620 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d\" (UID: \"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" Nov 02 23:30:00.426696 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.426673 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q848\" (UniqueName: \"kubernetes.io/projected/88fc9bc8-8e69-4a72-887e-c3612ce0055f-kube-api-access-2q848\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh\" (UID: \"88fc9bc8-8e69-4a72-887e-c3612ce0055f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" Nov 02 23:30:00.426905 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.426707 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fc9bc8-8e69-4a72-887e-c3612ce0055f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh\" (UID: \"88fc9bc8-8e69-4a72-887e-c3612ce0055f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" Nov 02 23:30:00.426905 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.426734 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894370-jmk5t\" (UID: \"c67b7984-ede5-4e8f-b266-34cd84c7f2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" Nov 02 23:30:00.426905 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.426798 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2xxw\" (UniqueName: \"kubernetes.io/projected/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-kube-api-access-j2xxw\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d\" (UID: \"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" Nov 02 23:30:00.427069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.426937 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7025314-94d4-4994-a385-eae79b919e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78\" (UID: \"e7025314-94d4-4994-a385-eae79b919e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" Nov 02 23:30:00.427069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.426964 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ms57\" (UniqueName: \"kubernetes.io/projected/e7025314-94d4-4994-a385-eae79b919e64-kube-api-access-2ms57\") pod \"maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78\" (UID: \"e7025314-94d4-4994-a385-eae79b919e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" Nov 02 23:30:00.427069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.426996 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2da10-0859-46c8-a214-e90ce9f44840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n\" (UID: \"8fc2da10-0859-46c8-a214-e90ce9f44840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" Nov 02 23:30:00.427069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.427040 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nghv\" (UniqueName: \"kubernetes.io/projected/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-kube-api-access-9nghv\") pod \"maas-api-key-cleanup-29894370-jmk5t\" (UID: \"c67b7984-ede5-4e8f-b266-34cd84c7f2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" Nov 02 23:30:00.427285 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.427087 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a900285e-ba2c-4da2-906b-b8c8f586441c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g\" (UID: \"a900285e-ba2c-4da2-906b-b8c8f586441c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" Nov 02 23:30:00.427285 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.427119 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p77f\" (UniqueName: \"kubernetes.io/projected/a900285e-ba2c-4da2-906b-b8c8f586441c-kube-api-access-2p77f\") pod \"maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g\" (UID: \"a900285e-ba2c-4da2-906b-b8c8f586441c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" Nov 02 23:30:00.427469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.427444 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d\" (UID: \"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" Nov 02 23:30:00.427785 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.427760 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fc9bc8-8e69-4a72-887e-c3612ce0055f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh\" (UID: \"88fc9bc8-8e69-4a72-887e-c3612ce0055f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" Nov 02 23:30:00.427883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.427864 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894370-jmk5t\" (UID: \"c67b7984-ede5-4e8f-b266-34cd84c7f2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" Nov 02 23:30:00.428123 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.428104 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2da10-0859-46c8-a214-e90ce9f44840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n\" (UID: \"8fc2da10-0859-46c8-a214-e90ce9f44840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" Nov 02 23:30:00.428624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.428585 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7025314-94d4-4994-a385-eae79b919e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78\" (UID: \"e7025314-94d4-4994-a385-eae79b919e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" Nov 02 23:30:00.428695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.428664 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a900285e-ba2c-4da2-906b-b8c8f586441c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g\" (UID: \"a900285e-ba2c-4da2-906b-b8c8f586441c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" Nov 02 23:30:00.429238 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.429205 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sxjk\" (UniqueName: \"kubernetes.io/projected/8fc2da10-0859-46c8-a214-e90ce9f44840-kube-api-access-9sxjk\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n\" (UID: \"8fc2da10-0859-46c8-a214-e90ce9f44840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" Nov 02 23:30:00.479257 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.479229 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" Nov 02 23:30:00.481100 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.481064 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2xxw\" (UniqueName: \"kubernetes.io/projected/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-kube-api-access-j2xxw\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d\" (UID: \"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" Nov 02 23:30:00.481902 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.481446 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nghv\" (UniqueName: \"kubernetes.io/projected/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-kube-api-access-9nghv\") pod \"maas-api-key-cleanup-29894370-jmk5t\" (UID: \"c67b7984-ede5-4e8f-b266-34cd84c7f2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" Nov 02 23:30:00.481902 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.481598 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q848\" (UniqueName: \"kubernetes.io/projected/88fc9bc8-8e69-4a72-887e-c3612ce0055f-kube-api-access-2q848\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh\" (UID: \"88fc9bc8-8e69-4a72-887e-c3612ce0055f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" Nov 02 23:30:00.482126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.482102 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p77f\" (UniqueName: \"kubernetes.io/projected/a900285e-ba2c-4da2-906b-b8c8f586441c-kube-api-access-2p77f\") pod \"maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g\" (UID: \"a900285e-ba2c-4da2-906b-b8c8f586441c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" Nov 02 23:30:00.482780 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.482756 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ms57\" (UniqueName: \"kubernetes.io/projected/e7025314-94d4-4994-a385-eae79b919e64-kube-api-access-2ms57\") pod \"maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78\" (UID: \"e7025314-94d4-4994-a385-eae79b919e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" Nov 02 23:30:00.482850 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.482817 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sxjk\" (UniqueName: \"kubernetes.io/projected/8fc2da10-0859-46c8-a214-e90ce9f44840-kube-api-access-9sxjk\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n\" (UID: \"8fc2da10-0859-46c8-a214-e90ce9f44840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" Nov 02 23:30:00.486923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.486900 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" Nov 02 23:30:00.493025 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.493009 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" Nov 02 23:30:00.528186 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.525899 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" Nov 02 23:30:00.534963 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.534928 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" Nov 02 23:30:00.543660 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.543421 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" Nov 02 23:30:00.553682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.553308 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" Nov 02 23:30:00.558189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.557947 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" Nov 02 23:30:00.859594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.859568 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x"] Nov 02 23:30:00.861684 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:30:00.861653 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64237b78_b4e0_42c6_b9e6_3f75e7710ff1.slice/crio-96432330962bf565a166731032cb02a48129874251de455ff8c59d2e974e85ce WatchSource:0}: Error finding container 96432330962bf565a166731032cb02a48129874251de455ff8c59d2e974e85ce: Status 404 returned error can't find the container with id 96432330962bf565a166731032cb02a48129874251de455ff8c59d2e974e85ce Nov 02 23:30:00.905794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.905768 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb"] Nov 02 23:30:00.907987 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:30:00.907928 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0498fc_b7d3_4b8c_a83f_0d08ed968c8f.slice/crio-19d4cba3b5d68cf5ed849e2a7371dc2c4cfb0cc3135ad3d4f3d34af45173f220 WatchSource:0}: Error finding container 19d4cba3b5d68cf5ed849e2a7371dc2c4cfb0cc3135ad3d4f3d34af45173f220: Status 404 returned error can't find the container with id 19d4cba3b5d68cf5ed849e2a7371dc2c4cfb0cc3135ad3d4f3d34af45173f220 Nov 02 23:30:00.911140 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.911118 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d"] Nov 02 23:30:00.913091 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:30:00.913064 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4d1076_7133_4df6_8b4b_7a61ff4bdb3c.slice/crio-0ae05e286d90a2d640ae08324d155c6177b1f8587a6bf797f462f372497d7d68 WatchSource:0}: Error finding container 0ae05e286d90a2d640ae08324d155c6177b1f8587a6bf797f462f372497d7d68: Status 404 returned error can't find the container with id 0ae05e286d90a2d640ae08324d155c6177b1f8587a6bf797f462f372497d7d68 Nov 02 23:30:00.928916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.928890 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" event={"ID":"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f","Type":"ContainerStarted","Data":"19d4cba3b5d68cf5ed849e2a7371dc2c4cfb0cc3135ad3d4f3d34af45173f220"} Nov 02 23:30:00.930232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.930171 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" event={"ID":"64237b78-b4e0-42c6-b9e6-3f75e7710ff1","Type":"ContainerStarted","Data":"96432330962bf565a166731032cb02a48129874251de455ff8c59d2e974e85ce"} Nov 02 23:30:00.931541 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.931505 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" event={"ID":"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c","Type":"ContainerStarted","Data":"0ae05e286d90a2d640ae08324d155c6177b1f8587a6bf797f462f372497d7d68"} Nov 02 23:30:00.956550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.956528 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh"] Nov 02 23:30:00.959357 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:30:00.959334 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fc9bc8_8e69_4a72_887e_c3612ce0055f.slice/crio-a7374fd286aa0c826bf2106cde30c22487f73e5f167034ce12a3fc66fbc1219c WatchSource:0}: Error finding container a7374fd286aa0c826bf2106cde30c22487f73e5f167034ce12a3fc66fbc1219c: Status 404 returned error can't find the container with id a7374fd286aa0c826bf2106cde30c22487f73e5f167034ce12a3fc66fbc1219c Nov 02 23:30:00.985040 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.985017 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t"] Nov 02 23:30:00.992968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:00.992949 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g"] Nov 02 23:30:00.994807 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:30:00.994781 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67b7984_ede5_4e8f_b266_34cd84c7f2c1.slice/crio-3ddc7a0e71d514e15a67a475178c3f84b86269dd67c91707b5bc2b07173a0220 WatchSource:0}: Error finding container 3ddc7a0e71d514e15a67a475178c3f84b86269dd67c91707b5bc2b07173a0220: Status 404 returned error can't find the container with id 3ddc7a0e71d514e15a67a475178c3f84b86269dd67c91707b5bc2b07173a0220 Nov 02 23:30:00.995038 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:30:00.995007 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda900285e_ba2c_4da2_906b_b8c8f586441c.slice/crio-d6f15236b3aa5bcf63f1a097fccb5c4f987dd405a3bdbe08d0e399c5553832a3 WatchSource:0}: Error finding container d6f15236b3aa5bcf63f1a097fccb5c4f987dd405a3bdbe08d0e399c5553832a3: Status 404 returned error can't find the container with id d6f15236b3aa5bcf63f1a097fccb5c4f987dd405a3bdbe08d0e399c5553832a3 Nov 02 23:30:01.006812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.006785 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n"] Nov 02 23:30:01.013503 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:30:01.013462 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc2da10_0859_46c8_a214_e90ce9f44840.slice/crio-b51587f1839749da2f4ea6e5d479002d1a86efd9491c688e9ced9f39df83291b WatchSource:0}: Error finding container b51587f1839749da2f4ea6e5d479002d1a86efd9491c688e9ced9f39df83291b: Status 404 returned error can't find the container with id b51587f1839749da2f4ea6e5d479002d1a86efd9491c688e9ced9f39df83291b Nov 02 23:30:01.019444 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.019419 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78"] Nov 02 23:30:01.022200 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:30:01.022175 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7025314_94d4_4994_a385_eae79b919e64.slice/crio-1b7b872a1cdd15e5ed985b5bf719f22d7a3caea076868b8232466fec308411b9 WatchSource:0}: Error finding container 1b7b872a1cdd15e5ed985b5bf719f22d7a3caea076868b8232466fec308411b9: Status 404 returned error can't find the container with id 1b7b872a1cdd15e5ed985b5bf719f22d7a3caea076868b8232466fec308411b9 Nov 02 23:30:01.941628 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.941594 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" event={"ID":"8fc2da10-0859-46c8-a214-e90ce9f44840","Type":"ContainerStarted","Data":"cc7e5e3f5f9e70e2091b395a4be7154a1165f621d32548fa53d10f75d56a80b8"} Nov 02 23:30:01.942033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.941655 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" event={"ID":"8fc2da10-0859-46c8-a214-e90ce9f44840","Type":"ContainerStarted","Data":"b51587f1839749da2f4ea6e5d479002d1a86efd9491c688e9ced9f39df83291b"} Nov 02 23:30:01.955053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.954190 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" event={"ID":"e7025314-94d4-4994-a385-eae79b919e64","Type":"ContainerStarted","Data":"4c1e93f2ffc58a04ccd4c57aae7b3767d7b42bff138a3be027c3566b33a9ed8c"} Nov 02 23:30:01.955053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.954300 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" event={"ID":"e7025314-94d4-4994-a385-eae79b919e64","Type":"ContainerStarted","Data":"1b7b872a1cdd15e5ed985b5bf719f22d7a3caea076868b8232466fec308411b9"} Nov 02 23:30:01.957326 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.957291 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" event={"ID":"c67b7984-ede5-4e8f-b266-34cd84c7f2c1","Type":"ContainerStarted","Data":"77948e23d2a468dfa39ee57397aa80e57020012b660e55255fd8530bc442e532"} Nov 02 23:30:01.957429 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.957347 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" event={"ID":"c67b7984-ede5-4e8f-b266-34cd84c7f2c1","Type":"ContainerStarted","Data":"3ddc7a0e71d514e15a67a475178c3f84b86269dd67c91707b5bc2b07173a0220"} Nov 02 23:30:01.959037 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.959014 5672 generic.go:356] "Generic (PLEG): container finished" podID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerID="6cbbe94c076fd4d32e8417f3566a91c6f6655e37b0a70c5b63310573530656ea" exitCode=60 Nov 02 23:30:01.959164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.959116 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" event={"ID":"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f","Type":"ContainerDied","Data":"6cbbe94c076fd4d32e8417f3566a91c6f6655e37b0a70c5b63310573530656ea"} Nov 02 23:30:01.959493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.959300 5672 scope.go:122] "RemoveContainer" containerID="6cbbe94c076fd4d32e8417f3566a91c6f6655e37b0a70c5b63310573530656ea" Nov 02 23:30:01.961095 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.961065 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" event={"ID":"88fc9bc8-8e69-4a72-887e-c3612ce0055f","Type":"ContainerStarted","Data":"a859814ddc30991b868097028cd6b309c8e83a2eaaf275fae14fff5acbe6d6c9"} Nov 02 23:30:01.961218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.961117 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" event={"ID":"88fc9bc8-8e69-4a72-887e-c3612ce0055f","Type":"ContainerStarted","Data":"a7374fd286aa0c826bf2106cde30c22487f73e5f167034ce12a3fc66fbc1219c"} Nov 02 23:30:01.963327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.963302 5672 generic.go:356] "Generic (PLEG): container finished" podID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerID="58a6073def13837663ffa5604684afce6ce3b9a3d5f38b0a93abf19283676a48" exitCode=60 Nov 02 23:30:01.963419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.963375 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" event={"ID":"64237b78-b4e0-42c6-b9e6-3f75e7710ff1","Type":"ContainerDied","Data":"58a6073def13837663ffa5604684afce6ce3b9a3d5f38b0a93abf19283676a48"} Nov 02 23:30:01.963639 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.963616 5672 scope.go:122] "RemoveContainer" containerID="58a6073def13837663ffa5604684afce6ce3b9a3d5f38b0a93abf19283676a48" Nov 02 23:30:01.964973 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.964954 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" event={"ID":"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c","Type":"ContainerStarted","Data":"f4e4efed281477853d916f115c23c98dd9b4185d48d828ad76eab6b4120d6412"} Nov 02 23:30:01.966547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.966525 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" event={"ID":"a900285e-ba2c-4da2-906b-b8c8f586441c","Type":"ContainerStarted","Data":"e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b"} Nov 02 23:30:01.966650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.966560 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" event={"ID":"a900285e-ba2c-4da2-906b-b8c8f586441c","Type":"ContainerStarted","Data":"d6f15236b3aa5bcf63f1a097fccb5c4f987dd405a3bdbe08d0e399c5553832a3"} Nov 02 23:30:01.966727 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:01.966715 5672 scope.go:122] "RemoveContainer" containerID="e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b" Nov 02 23:30:02.071757 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.071710 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" podStartSLOduration=2.071696247 podStartE2EDuration="2.071696247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.068440511 +0000 UTC m=+2195.475752339" watchObservedRunningTime="2026-11-02 23:30:02.071696247 +0000 UTC m=+2195.479008076" Nov 02 23:30:02.289328 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.289276 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" podStartSLOduration=2.289261962 podStartE2EDuration="2.289261962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.286381748 +0000 UTC m=+2195.693693578" watchObservedRunningTime="2026-11-02 23:30:02.289261962 +0000 UTC m=+2195.696573804" Nov 02 23:30:02.396416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.396372 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" podStartSLOduration=2.396359601 podStartE2EDuration="2.396359601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.39424543 +0000 UTC m=+2195.801557260" watchObservedRunningTime="2026-11-02 23:30:02.396359601 +0000 UTC m=+2195.803671454" Nov 02 23:30:02.500475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.500420 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" podStartSLOduration=2.5004037439999998 podStartE2EDuration="2.500403744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.499819691 +0000 UTC m=+2195.907131501" watchObservedRunningTime="2026-11-02 23:30:02.500403744 +0000 UTC m=+2195.907715574" Nov 02 23:30:02.606924 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.606865 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" podStartSLOduration=2.606849354 podStartE2EDuration="2.606849354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:02.606228247 +0000 UTC m=+2196.013540075" watchObservedRunningTime="2026-11-02 23:30:02.606849354 +0000 UTC m=+2196.014161186" Nov 02 23:30:02.975865 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.975835 5672 generic.go:356] "Generic (PLEG): container finished" podID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerID="a859814ddc30991b868097028cd6b309c8e83a2eaaf275fae14fff5acbe6d6c9" exitCode=60 Nov 02 23:30:02.976212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.975913 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" event={"ID":"88fc9bc8-8e69-4a72-887e-c3612ce0055f","Type":"ContainerDied","Data":"a859814ddc30991b868097028cd6b309c8e83a2eaaf275fae14fff5acbe6d6c9"} Nov 02 23:30:02.976279 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.976245 5672 scope.go:122] "RemoveContainer" containerID="a859814ddc30991b868097028cd6b309c8e83a2eaaf275fae14fff5acbe6d6c9" Nov 02 23:30:02.978079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.978057 5672 generic.go:356] "Generic (PLEG): container finished" podID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerID="48e56f6b6054ea6fbe5a2b6c6eb1c1daf4fa99ea3c457692d56ad26523f2b55a" exitCode=60 Nov 02 23:30:02.978191 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.978140 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" event={"ID":"64237b78-b4e0-42c6-b9e6-3f75e7710ff1","Type":"ContainerDied","Data":"48e56f6b6054ea6fbe5a2b6c6eb1c1daf4fa99ea3c457692d56ad26523f2b55a"} Nov 02 23:30:02.978258 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.978208 5672 scope.go:122] "RemoveContainer" containerID="58a6073def13837663ffa5604684afce6ce3b9a3d5f38b0a93abf19283676a48" Nov 02 23:30:02.978559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.978530 5672 scope.go:122] "RemoveContainer" containerID="48e56f6b6054ea6fbe5a2b6c6eb1c1daf4fa99ea3c457692d56ad26523f2b55a" Nov 02 23:30:02.979484 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:02.979453 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x_odh-ai-gateway-infra(64237b78-b4e0-42c6-b9e6-3f75e7710ff1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" Nov 02 23:30:02.980186 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.980143 5672 generic.go:356] "Generic (PLEG): container finished" podID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerID="f4e4efed281477853d916f115c23c98dd9b4185d48d828ad76eab6b4120d6412" exitCode=60 Nov 02 23:30:02.980290 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.980235 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" event={"ID":"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c","Type":"ContainerDied","Data":"f4e4efed281477853d916f115c23c98dd9b4185d48d828ad76eab6b4120d6412"} Nov 02 23:30:02.980497 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.980478 5672 scope.go:122] "RemoveContainer" containerID="f4e4efed281477853d916f115c23c98dd9b4185d48d828ad76eab6b4120d6412" Nov 02 23:30:02.982428 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.982407 5672 generic.go:356] "Generic (PLEG): container finished" podID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerID="e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b" exitCode=7 Nov 02 23:30:02.982428 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.982424 5672 generic.go:356] "Generic (PLEG): container finished" podID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerID="5b06b386302a1dd7daf6049a45ed9bb6f118470644c8dfbecdd12f9e637b79bd" exitCode=7 Nov 02 23:30:02.982630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.982446 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" event={"ID":"a900285e-ba2c-4da2-906b-b8c8f586441c","Type":"ContainerDied","Data":"e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b"} Nov 02 23:30:02.982630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.982504 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" event={"ID":"a900285e-ba2c-4da2-906b-b8c8f586441c","Type":"ContainerDied","Data":"5b06b386302a1dd7daf6049a45ed9bb6f118470644c8dfbecdd12f9e637b79bd"} Nov 02 23:30:02.982789 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.982725 5672 scope.go:122] "RemoveContainer" containerID="5b06b386302a1dd7daf6049a45ed9bb6f118470644c8dfbecdd12f9e637b79bd" Nov 02 23:30:02.982988 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:02.982961 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g_odh-ai-gateway-infra(a900285e-ba2c-4da2-906b-b8c8f586441c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" Nov 02 23:30:02.984438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.984402 5672 generic.go:356] "Generic (PLEG): container finished" podID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerID="cc7e5e3f5f9e70e2091b395a4be7154a1165f621d32548fa53d10f75d56a80b8" exitCode=60 Nov 02 23:30:02.984553 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.984489 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" event={"ID":"8fc2da10-0859-46c8-a214-e90ce9f44840","Type":"ContainerDied","Data":"cc7e5e3f5f9e70e2091b395a4be7154a1165f621d32548fa53d10f75d56a80b8"} Nov 02 23:30:02.984682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.984666 5672 scope.go:122] "RemoveContainer" containerID="cc7e5e3f5f9e70e2091b395a4be7154a1165f621d32548fa53d10f75d56a80b8" Nov 02 23:30:02.986247 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.986224 5672 generic.go:356] "Generic (PLEG): container finished" podID="e7025314-94d4-4994-a385-eae79b919e64" containerID="4c1e93f2ffc58a04ccd4c57aae7b3767d7b42bff138a3be027c3566b33a9ed8c" exitCode=60 Nov 02 23:30:02.986358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.986302 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" event={"ID":"e7025314-94d4-4994-a385-eae79b919e64","Type":"ContainerDied","Data":"4c1e93f2ffc58a04ccd4c57aae7b3767d7b42bff138a3be027c3566b33a9ed8c"} Nov 02 23:30:02.986469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.986455 5672 scope.go:122] "RemoveContainer" containerID="4c1e93f2ffc58a04ccd4c57aae7b3767d7b42bff138a3be027c3566b33a9ed8c" Nov 02 23:30:02.988301 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.988279 5672 generic.go:356] "Generic (PLEG): container finished" podID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerID="77948e23d2a468dfa39ee57397aa80e57020012b660e55255fd8530bc442e532" exitCode=60 Nov 02 23:30:02.988475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.988431 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" event={"ID":"c67b7984-ede5-4e8f-b266-34cd84c7f2c1","Type":"ContainerDied","Data":"77948e23d2a468dfa39ee57397aa80e57020012b660e55255fd8530bc442e532"} Nov 02 23:30:02.989079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.989046 5672 scope.go:122] "RemoveContainer" containerID="77948e23d2a468dfa39ee57397aa80e57020012b660e55255fd8530bc442e532" Nov 02 23:30:02.995251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.995231 5672 generic.go:356] "Generic (PLEG): container finished" podID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerID="67755268480ab72ecda73578be36a5ce1e090b640bb128977308b010ed63de49" exitCode=60 Nov 02 23:30:02.995309 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.995283 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" event={"ID":"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f","Type":"ContainerDied","Data":"67755268480ab72ecda73578be36a5ce1e090b640bb128977308b010ed63de49"} Nov 02 23:30:02.995673 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:02.995555 5672 scope.go:122] "RemoveContainer" containerID="67755268480ab72ecda73578be36a5ce1e090b640bb128977308b010ed63de49" Nov 02 23:30:02.995780 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:02.995746 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb_odh-ai-gateway-infra(8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" Nov 02 23:30:03.021073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:03.021054 5672 scope.go:122] "RemoveContainer" containerID="e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b" Nov 02 23:30:03.042943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:03.042920 5672 scope.go:122] "RemoveContainer" containerID="e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b" Nov 02 23:30:03.043284 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:03.043263 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b\": container with ID starting with e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b not found: ID does not exist" containerID="e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b" Nov 02 23:30:03.043358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:03.043290 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b"} err="failed to get container status \"e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b\": rpc error: code = NotFound desc = could not find container \"e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b\": container with ID starting with e06d4a44f7e4eeddd1684158bc1b61538c4dc1d9cad59b2427dec8d832b92c3b not found: ID does not exist" Nov 02 23:30:03.043358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:03.043310 5672 scope.go:122] "RemoveContainer" containerID="6cbbe94c076fd4d32e8417f3566a91c6f6655e37b0a70c5b63310573530656ea" Nov 02 23:30:04.006209 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:04.006176 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" event={"ID":"88fc9bc8-8e69-4a72-887e-c3612ce0055f","Type":"ContainerStarted","Data":"04986fd9d8b2212b60e166cf2f8fb97eb48c0bcb89e456d76f8e54a91efae1ae"} Nov 02 23:30:04.008380 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:04.008354 5672 scope.go:122] "RemoveContainer" containerID="48e56f6b6054ea6fbe5a2b6c6eb1c1daf4fa99ea3c457692d56ad26523f2b55a" Nov 02 23:30:04.008645 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:04.008615 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x_odh-ai-gateway-infra(64237b78-b4e0-42c6-b9e6-3f75e7710ff1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" Nov 02 23:30:04.009914 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:04.009887 5672 generic.go:356] "Generic (PLEG): container finished" podID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerID="95b2711d7734ed9a956961d7298ebb46ebdcf9fc46b4467c3df7423cea45a74d" exitCode=60 Nov 02 23:30:04.010018 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:04.009959 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" event={"ID":"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c","Type":"ContainerDied","Data":"95b2711d7734ed9a956961d7298ebb46ebdcf9fc46b4467c3df7423cea45a74d"} Nov 02 23:30:04.010018 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:04.010003 5672 scope.go:122] "RemoveContainer" containerID="f4e4efed281477853d916f115c23c98dd9b4185d48d828ad76eab6b4120d6412" Nov 02 23:30:04.010222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:04.010208 5672 scope.go:122] "RemoveContainer" containerID="95b2711d7734ed9a956961d7298ebb46ebdcf9fc46b4467c3df7423cea45a74d" Nov 02 23:30:04.010905 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:04.010875 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d_odh-ai-gateway-infra(0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" Nov 02 23:30:04.021999 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:04.021980 5672 scope.go:122] "RemoveContainer" containerID="5b06b386302a1dd7daf6049a45ed9bb6f118470644c8dfbecdd12f9e637b79bd" Nov 02 23:30:04.022215 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:04.022194 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g_odh-ai-gateway-infra(a900285e-ba2c-4da2-906b-b8c8f586441c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" Nov 02 23:30:04.023847 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:04.023816 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" event={"ID":"8fc2da10-0859-46c8-a214-e90ce9f44840","Type":"ContainerStarted","Data":"ef934187df8d54447d57804df20419901de27639e006c8b1bbdae9c56a45d656"} Nov 02 23:30:04.028008 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:04.027983 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" event={"ID":"e7025314-94d4-4994-a385-eae79b919e64","Type":"ContainerStarted","Data":"457cda35552c32e6ff9ef01075472e16a3159509148b887fb36bb3e697d5f72b"} Nov 02 23:30:04.031623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:04.031597 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" event={"ID":"c67b7984-ede5-4e8f-b266-34cd84c7f2c1","Type":"ContainerStarted","Data":"8aa1408039986fca4c4847d36c63fe5cf88b77aa30f4cde100438ca7ac71d782"} Nov 02 23:30:04.033664 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:04.033650 5672 scope.go:122] "RemoveContainer" containerID="67755268480ab72ecda73578be36a5ce1e090b640bb128977308b010ed63de49" Nov 02 23:30:04.033811 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:04.033796 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb_odh-ai-gateway-infra(8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" Nov 02 23:30:05.042823 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.042790 5672 generic.go:356] "Generic (PLEG): container finished" podID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerID="04986fd9d8b2212b60e166cf2f8fb97eb48c0bcb89e456d76f8e54a91efae1ae" exitCode=60 Nov 02 23:30:05.043311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.042872 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" event={"ID":"88fc9bc8-8e69-4a72-887e-c3612ce0055f","Type":"ContainerDied","Data":"04986fd9d8b2212b60e166cf2f8fb97eb48c0bcb89e456d76f8e54a91efae1ae"} Nov 02 23:30:05.043311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.043173 5672 scope.go:122] "RemoveContainer" containerID="04986fd9d8b2212b60e166cf2f8fb97eb48c0bcb89e456d76f8e54a91efae1ae" Nov 02 23:30:05.043311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.043262 5672 scope.go:122] "RemoveContainer" containerID="a859814ddc30991b868097028cd6b309c8e83a2eaaf275fae14fff5acbe6d6c9" Nov 02 23:30:05.043490 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:05.043471 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh_odh-ai-gateway-infra(88fc9bc8-8e69-4a72-887e-c3612ce0055f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" Nov 02 23:30:05.045416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.045399 5672 scope.go:122] "RemoveContainer" containerID="95b2711d7734ed9a956961d7298ebb46ebdcf9fc46b4467c3df7423cea45a74d" Nov 02 23:30:05.045634 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:05.045614 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d_odh-ai-gateway-infra(0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" Nov 02 23:30:05.047159 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.047124 5672 generic.go:356] "Generic (PLEG): container finished" podID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerID="ef934187df8d54447d57804df20419901de27639e006c8b1bbdae9c56a45d656" exitCode=60 Nov 02 23:30:05.047256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.047214 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" event={"ID":"8fc2da10-0859-46c8-a214-e90ce9f44840","Type":"ContainerDied","Data":"ef934187df8d54447d57804df20419901de27639e006c8b1bbdae9c56a45d656"} Nov 02 23:30:05.047492 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.047476 5672 scope.go:122] "RemoveContainer" containerID="ef934187df8d54447d57804df20419901de27639e006c8b1bbdae9c56a45d656" Nov 02 23:30:05.047736 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:05.047717 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n_odh-ai-gateway-infra(8fc2da10-0859-46c8-a214-e90ce9f44840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" Nov 02 23:30:05.049547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.049522 5672 generic.go:356] "Generic (PLEG): container finished" podID="e7025314-94d4-4994-a385-eae79b919e64" containerID="457cda35552c32e6ff9ef01075472e16a3159509148b887fb36bb3e697d5f72b" exitCode=60 Nov 02 23:30:05.049626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.049555 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" event={"ID":"e7025314-94d4-4994-a385-eae79b919e64","Type":"ContainerDied","Data":"457cda35552c32e6ff9ef01075472e16a3159509148b887fb36bb3e697d5f72b"} Nov 02 23:30:05.049813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.049796 5672 scope.go:122] "RemoveContainer" containerID="457cda35552c32e6ff9ef01075472e16a3159509148b887fb36bb3e697d5f72b" Nov 02 23:30:05.050019 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:05.049999 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78_odh-ai-gateway-infra(e7025314-94d4-4994-a385-eae79b919e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" podUID="e7025314-94d4-4994-a385-eae79b919e64" Nov 02 23:30:05.051647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.051616 5672 generic.go:356] "Generic (PLEG): container finished" podID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerID="8aa1408039986fca4c4847d36c63fe5cf88b77aa30f4cde100438ca7ac71d782" exitCode=60 Nov 02 23:30:05.051772 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.051669 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" event={"ID":"c67b7984-ede5-4e8f-b266-34cd84c7f2c1","Type":"ContainerDied","Data":"8aa1408039986fca4c4847d36c63fe5cf88b77aa30f4cde100438ca7ac71d782"} Nov 02 23:30:05.055181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.051922 5672 scope.go:122] "RemoveContainer" containerID="8aa1408039986fca4c4847d36c63fe5cf88b77aa30f4cde100438ca7ac71d782" Nov 02 23:30:05.055181 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:05.052109 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894370-jmk5t_odh-ai-gateway-infra(c67b7984-ede5-4e8f-b266-34cd84c7f2c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" Nov 02 23:30:05.056459 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.056441 5672 scope.go:122] "RemoveContainer" containerID="cc7e5e3f5f9e70e2091b395a4be7154a1165f621d32548fa53d10f75d56a80b8" Nov 02 23:30:05.068810 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.068787 5672 scope.go:122] "RemoveContainer" containerID="4c1e93f2ffc58a04ccd4c57aae7b3767d7b42bff138a3be027c3566b33a9ed8c" Nov 02 23:30:05.082373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:05.082289 5672 scope.go:122] "RemoveContainer" containerID="77948e23d2a468dfa39ee57397aa80e57020012b660e55255fd8530bc442e532" Nov 02 23:30:06.061040 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:06.061005 5672 scope.go:122] "RemoveContainer" containerID="04986fd9d8b2212b60e166cf2f8fb97eb48c0bcb89e456d76f8e54a91efae1ae" Nov 02 23:30:06.061520 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:06.061324 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh_odh-ai-gateway-infra(88fc9bc8-8e69-4a72-887e-c3612ce0055f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" Nov 02 23:30:06.062761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:06.062733 5672 scope.go:122] "RemoveContainer" containerID="ef934187df8d54447d57804df20419901de27639e006c8b1bbdae9c56a45d656" Nov 02 23:30:06.063030 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:06.062989 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n_odh-ai-gateway-infra(8fc2da10-0859-46c8-a214-e90ce9f44840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" Nov 02 23:30:06.064619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:06.064586 5672 scope.go:122] "RemoveContainer" containerID="457cda35552c32e6ff9ef01075472e16a3159509148b887fb36bb3e697d5f72b" Nov 02 23:30:06.064869 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:06.064842 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78_odh-ai-gateway-infra(e7025314-94d4-4994-a385-eae79b919e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" podUID="e7025314-94d4-4994-a385-eae79b919e64" Nov 02 23:30:06.066257 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:06.066238 5672 scope.go:122] "RemoveContainer" containerID="8aa1408039986fca4c4847d36c63fe5cf88b77aa30f4cde100438ca7ac71d782" Nov 02 23:30:06.066482 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:06.066465 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894370-jmk5t_odh-ai-gateway-infra(c67b7984-ede5-4e8f-b266-34cd84c7f2c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" Nov 02 23:30:13.706561 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:13.706532 5672 scope.go:122] "RemoveContainer" containerID="48e56f6b6054ea6fbe5a2b6c6eb1c1daf4fa99ea3c457692d56ad26523f2b55a" Nov 02 23:30:14.121044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:14.121008 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" event={"ID":"64237b78-b4e0-42c6-b9e6-3f75e7710ff1","Type":"ContainerStarted","Data":"589bc213c15a70f773453a4de78604b0fa55aebc29eb03b552f861e69d34fbf0"} Nov 02 23:30:14.228280 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:14.228229 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" podStartSLOduration=14.228216029 podStartE2EDuration="14.228216029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:14.226167358 +0000 UTC m=+2207.633479179" watchObservedRunningTime="2026-11-02 23:30:14.228216029 +0000 UTC m=+2207.635527858" Nov 02 23:30:14.706173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:14.706112 5672 scope.go:122] "RemoveContainer" containerID="8aa1408039986fca4c4847d36c63fe5cf88b77aa30f4cde100438ca7ac71d782" Nov 02 23:30:15.130322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:15.130279 5672 generic.go:356] "Generic (PLEG): container finished" podID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerID="589bc213c15a70f773453a4de78604b0fa55aebc29eb03b552f861e69d34fbf0" exitCode=60 Nov 02 23:30:15.130815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:15.130373 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" event={"ID":"64237b78-b4e0-42c6-b9e6-3f75e7710ff1","Type":"ContainerDied","Data":"589bc213c15a70f773453a4de78604b0fa55aebc29eb03b552f861e69d34fbf0"} Nov 02 23:30:15.130815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:15.130442 5672 scope.go:122] "RemoveContainer" containerID="48e56f6b6054ea6fbe5a2b6c6eb1c1daf4fa99ea3c457692d56ad26523f2b55a" Nov 02 23:30:15.130815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:15.130786 5672 scope.go:122] "RemoveContainer" containerID="589bc213c15a70f773453a4de78604b0fa55aebc29eb03b552f861e69d34fbf0" Nov 02 23:30:15.131117 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:15.131088 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x_odh-ai-gateway-infra(64237b78-b4e0-42c6-b9e6-3f75e7710ff1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" Nov 02 23:30:15.132582 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:15.132550 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" event={"ID":"c67b7984-ede5-4e8f-b266-34cd84c7f2c1","Type":"ContainerStarted","Data":"82bdf8938c27ef213a2f2d60bb80862d0313880366a27647b09c308f19bb3a8c"} Nov 02 23:30:15.237027 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:15.236987 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x"] Nov 02 23:30:16.143668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.143634 5672 generic.go:356] "Generic (PLEG): container finished" podID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerID="82bdf8938c27ef213a2f2d60bb80862d0313880366a27647b09c308f19bb3a8c" exitCode=60 Nov 02 23:30:16.144080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.143677 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" event={"ID":"c67b7984-ede5-4e8f-b266-34cd84c7f2c1","Type":"ContainerDied","Data":"82bdf8938c27ef213a2f2d60bb80862d0313880366a27647b09c308f19bb3a8c"} Nov 02 23:30:16.144080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.143754 5672 scope.go:122] "RemoveContainer" containerID="8aa1408039986fca4c4847d36c63fe5cf88b77aa30f4cde100438ca7ac71d782" Nov 02 23:30:16.144080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.144053 5672 scope.go:122] "RemoveContainer" containerID="82bdf8938c27ef213a2f2d60bb80862d0313880366a27647b09c308f19bb3a8c" Nov 02 23:30:16.144321 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:16.144301 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894370-jmk5t_odh-ai-gateway-infra(c67b7984-ede5-4e8f-b266-34cd84c7f2c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" Nov 02 23:30:16.294786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.294762 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" Nov 02 23:30:16.329603 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.329573 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-kube-api-access-tfw92\" (UniqueName: \"kubernetes.io/projected/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-kube-api-access-tfw92\") pod \"64237b78-b4e0-42c6-b9e6-3f75e7710ff1\" (UID: \"64237b78-b4e0-42c6-b9e6-3f75e7710ff1\") " Nov 02 23:30:16.329751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.329606 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-maas-api-service-ca\") pod \"64237b78-b4e0-42c6-b9e6-3f75e7710ff1\" (UID: \"64237b78-b4e0-42c6-b9e6-3f75e7710ff1\") " Nov 02 23:30:16.329991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.329965 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-maas-api-service-ca" pod "64237b78-b4e0-42c6-b9e6-3f75e7710ff1" (UID: "64237b78-b4e0-42c6-b9e6-3f75e7710ff1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:16.331767 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.331745 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-kube-api-access-tfw92" pod "64237b78-b4e0-42c6-b9e6-3f75e7710ff1" (UID: "64237b78-b4e0-42c6-b9e6-3f75e7710ff1"). InnerVolumeSpecName "kube-api-access-tfw92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:16.359929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.359893 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t"] Nov 02 23:30:16.431318 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.431249 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfw92\" (UniqueName: \"kubernetes.io/projected/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-kube-api-access-tfw92\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:16.431318 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.431281 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64237b78-b4e0-42c6-b9e6-3f75e7710ff1-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:16.709217 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.709112 5672 scope.go:122] "RemoveContainer" containerID="5b06b386302a1dd7daf6049a45ed9bb6f118470644c8dfbecdd12f9e637b79bd" Nov 02 23:30:16.709343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.709252 5672 scope.go:122] "RemoveContainer" containerID="67755268480ab72ecda73578be36a5ce1e090b640bb128977308b010ed63de49" Nov 02 23:30:16.709423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:16.709353 5672 scope.go:122] "RemoveContainer" containerID="457cda35552c32e6ff9ef01075472e16a3159509148b887fb36bb3e697d5f72b" Nov 02 23:30:17.157295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.157259 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" event={"ID":"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f","Type":"ContainerStarted","Data":"8cd4d5e0e7f44afe77a686a0d32621ea2409e34996e8af569434faef7aa24087"} Nov 02 23:30:17.159095 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.159074 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" Nov 02 23:30:17.159262 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.159094 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x" event={"ID":"64237b78-b4e0-42c6-b9e6-3f75e7710ff1","Type":"ContainerDied","Data":"96432330962bf565a166731032cb02a48129874251de455ff8c59d2e974e85ce"} Nov 02 23:30:17.159262 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.159188 5672 scope.go:122] "RemoveContainer" containerID="589bc213c15a70f773453a4de78604b0fa55aebc29eb03b552f861e69d34fbf0" Nov 02 23:30:17.163183 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.162471 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" event={"ID":"a900285e-ba2c-4da2-906b-b8c8f586441c","Type":"ContainerStarted","Data":"4ba15a58616e12046a85cb556d8832d29854f96970e259fc78227622713500e4"} Nov 02 23:30:17.164705 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.164683 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" event={"ID":"e7025314-94d4-4994-a385-eae79b919e64","Type":"ContainerStarted","Data":"0e72af92a7b91fa91ce01f41aab05ba6af163a769d0cbeee4b0fd250876d273e"} Nov 02 23:30:17.271178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.269506 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" podStartSLOduration=17.269488295 podStartE2EDuration="17.269488295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:17.268550648 +0000 UTC m=+2210.675862478" watchObservedRunningTime="2026-11-02 23:30:17.269488295 +0000 UTC m=+2210.676800126" Nov 02 23:30:17.315883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.315860 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" Nov 02 23:30:17.445808 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.445774 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-maas-api-service-ca\") pod \"c67b7984-ede5-4e8f-b266-34cd84c7f2c1\" (UID: \"c67b7984-ede5-4e8f-b266-34cd84c7f2c1\") " Nov 02 23:30:17.446000 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.445915 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-kube-api-access-9nghv\" (UniqueName: \"kubernetes.io/projected/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-kube-api-access-9nghv\") pod \"c67b7984-ede5-4e8f-b266-34cd84c7f2c1\" (UID: \"c67b7984-ede5-4e8f-b266-34cd84c7f2c1\") " Nov 02 23:30:17.446221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.446191 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-maas-api-service-ca" pod "c67b7984-ede5-4e8f-b266-34cd84c7f2c1" (UID: "c67b7984-ede5-4e8f-b266-34cd84c7f2c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:17.448036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.448005 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-kube-api-access-9nghv" pod "c67b7984-ede5-4e8f-b266-34cd84c7f2c1" (UID: "c67b7984-ede5-4e8f-b266-34cd84c7f2c1"). InnerVolumeSpecName "kube-api-access-9nghv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:17.547938 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.547856 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nghv\" (UniqueName: \"kubernetes.io/projected/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-kube-api-access-9nghv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:17.547938 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.547894 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67b7984-ede5-4e8f-b266-34cd84c7f2c1-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:17.586002 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.585935 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" podStartSLOduration=17.585918849 podStartE2EDuration="17.585918849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:17.584562191 +0000 UTC m=+2210.991874020" watchObservedRunningTime="2026-11-02 23:30:17.585918849 +0000 UTC m=+2210.993230681" Nov 02 23:30:17.691271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.691240 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x"] Nov 02 23:30:17.694938 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.694908 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ff336-29894370-j8q7x"] Nov 02 23:30:17.705423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.705399 5672 scope.go:122] "RemoveContainer" containerID="95b2711d7734ed9a956961d7298ebb46ebdcf9fc46b4467c3df7423cea45a74d" Nov 02 23:30:17.705528 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:17.705510 5672 scope.go:122] "RemoveContainer" containerID="ef934187df8d54447d57804df20419901de27639e006c8b1bbdae9c56a45d656" Nov 02 23:30:18.175840 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.175807 5672 generic.go:356] "Generic (PLEG): container finished" podID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerID="4ba15a58616e12046a85cb556d8832d29854f96970e259fc78227622713500e4" exitCode=60 Nov 02 23:30:18.176271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.175877 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" event={"ID":"a900285e-ba2c-4da2-906b-b8c8f586441c","Type":"ContainerDied","Data":"4ba15a58616e12046a85cb556d8832d29854f96970e259fc78227622713500e4"} Nov 02 23:30:18.176271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.175966 5672 scope.go:122] "RemoveContainer" containerID="5b06b386302a1dd7daf6049a45ed9bb6f118470644c8dfbecdd12f9e637b79bd" Nov 02 23:30:18.176389 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.176371 5672 scope.go:122] "RemoveContainer" containerID="4ba15a58616e12046a85cb556d8832d29854f96970e259fc78227622713500e4" Nov 02 23:30:18.176670 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:18.176646 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g_odh-ai-gateway-infra(a900285e-ba2c-4da2-906b-b8c8f586441c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" Nov 02 23:30:18.177996 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.177975 5672 generic.go:356] "Generic (PLEG): container finished" podID="e7025314-94d4-4994-a385-eae79b919e64" containerID="0e72af92a7b91fa91ce01f41aab05ba6af163a769d0cbeee4b0fd250876d273e" exitCode=60 Nov 02 23:30:18.178093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.178044 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" event={"ID":"e7025314-94d4-4994-a385-eae79b919e64","Type":"ContainerDied","Data":"0e72af92a7b91fa91ce01f41aab05ba6af163a769d0cbeee4b0fd250876d273e"} Nov 02 23:30:18.178424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.178408 5672 scope.go:122] "RemoveContainer" containerID="0e72af92a7b91fa91ce01f41aab05ba6af163a769d0cbeee4b0fd250876d273e" Nov 02 23:30:18.178664 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:18.178645 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78_odh-ai-gateway-infra(e7025314-94d4-4994-a385-eae79b919e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" podUID="e7025314-94d4-4994-a385-eae79b919e64" Nov 02 23:30:18.179351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.179335 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" Nov 02 23:30:18.179433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.179367 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t" event={"ID":"c67b7984-ede5-4e8f-b266-34cd84c7f2c1","Type":"ContainerDied","Data":"3ddc7a0e71d514e15a67a475178c3f84b86269dd67c91707b5bc2b07173a0220"} Nov 02 23:30:18.181336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.181312 5672 generic.go:356] "Generic (PLEG): container finished" podID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerID="8cd4d5e0e7f44afe77a686a0d32621ea2409e34996e8af569434faef7aa24087" exitCode=60 Nov 02 23:30:18.181447 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.181340 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" event={"ID":"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f","Type":"ContainerDied","Data":"8cd4d5e0e7f44afe77a686a0d32621ea2409e34996e8af569434faef7aa24087"} Nov 02 23:30:18.181637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.181622 5672 scope.go:122] "RemoveContainer" containerID="8cd4d5e0e7f44afe77a686a0d32621ea2409e34996e8af569434faef7aa24087" Nov 02 23:30:18.181827 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:18.181810 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb_odh-ai-gateway-infra(8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" Nov 02 23:30:18.189267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.189253 5672 scope.go:122] "RemoveContainer" containerID="457cda35552c32e6ff9ef01075472e16a3159509148b887fb36bb3e697d5f72b" Nov 02 23:30:18.201463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.201446 5672 scope.go:122] "RemoveContainer" containerID="82bdf8938c27ef213a2f2d60bb80862d0313880366a27647b09c308f19bb3a8c" Nov 02 23:30:18.209522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.209506 5672 scope.go:122] "RemoveContainer" containerID="67755268480ab72ecda73578be36a5ce1e090b640bb128977308b010ed63de49" Nov 02 23:30:18.276314 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.276289 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb"] Nov 02 23:30:18.383944 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.383915 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78"] Nov 02 23:30:18.388213 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.388188 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t"] Nov 02 23:30:18.393421 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.393363 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894370-jmk5t"] Nov 02 23:30:18.595134 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.595102 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g"] Nov 02 23:30:18.706374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.706282 5672 scope.go:122] "RemoveContainer" containerID="04986fd9d8b2212b60e166cf2f8fb97eb48c0bcb89e456d76f8e54a91efae1ae" Nov 02 23:30:18.714319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.714287 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" path="/var/lib/kubelet/pods/64237b78-b4e0-42c6-b9e6-3f75e7710ff1/volumes" Nov 02 23:30:18.714877 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:18.714860 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" path="/var/lib/kubelet/pods/c67b7984-ede5-4e8f-b266-34cd84c7f2c1/volumes" Nov 02 23:30:19.194889 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.194858 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" event={"ID":"88fc9bc8-8e69-4a72-887e-c3612ce0055f","Type":"ContainerStarted","Data":"c603e530a7b300cfb0e434f942053189396f555f7b9444b4f8892696e42a0d18"} Nov 02 23:30:19.196743 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.196720 5672 generic.go:356] "Generic (PLEG): container finished" podID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerID="1e9c17be5d09d0aa91d3a06719c291218e65117860d6d036eb6c51c0cf61df44" exitCode=60 Nov 02 23:30:19.196863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.196792 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" event={"ID":"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c","Type":"ContainerDied","Data":"1e9c17be5d09d0aa91d3a06719c291218e65117860d6d036eb6c51c0cf61df44"} Nov 02 23:30:19.196863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.196845 5672 scope.go:122] "RemoveContainer" containerID="95b2711d7734ed9a956961d7298ebb46ebdcf9fc46b4467c3df7423cea45a74d" Nov 02 23:30:19.197178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.197134 5672 scope.go:122] "RemoveContainer" containerID="1e9c17be5d09d0aa91d3a06719c291218e65117860d6d036eb6c51c0cf61df44" Nov 02 23:30:19.197445 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:19.197419 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d_odh-ai-gateway-infra(0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" Nov 02 23:30:19.200330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.200312 5672 generic.go:356] "Generic (PLEG): container finished" podID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerID="c6d5ab43f9832ed7c30f5c73f3d8ba9fa07fd9aeb0c847445de09471b715659d" exitCode=60 Nov 02 23:30:19.200435 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.200386 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" event={"ID":"8fc2da10-0859-46c8-a214-e90ce9f44840","Type":"ContainerDied","Data":"c6d5ab43f9832ed7c30f5c73f3d8ba9fa07fd9aeb0c847445de09471b715659d"} Nov 02 23:30:19.200762 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.200745 5672 scope.go:122] "RemoveContainer" containerID="c6d5ab43f9832ed7c30f5c73f3d8ba9fa07fd9aeb0c847445de09471b715659d" Nov 02 23:30:19.201022 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:19.201002 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n_odh-ai-gateway-infra(8fc2da10-0859-46c8-a214-e90ce9f44840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" Nov 02 23:30:19.212220 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.212197 5672 scope.go:122] "RemoveContainer" containerID="ef934187df8d54447d57804df20419901de27639e006c8b1bbdae9c56a45d656" Nov 02 23:30:19.377387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.377362 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" Nov 02 23:30:19.397405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.397380 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" Nov 02 23:30:19.431584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.431559 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" Nov 02 23:30:19.471483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.471456 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7025314-94d4-4994-a385-eae79b919e64-kube-api-access-2ms57\" (UniqueName: \"kubernetes.io/projected/e7025314-94d4-4994-a385-eae79b919e64-kube-api-access-2ms57\") pod \"e7025314-94d4-4994-a385-eae79b919e64\" (UID: \"e7025314-94d4-4994-a385-eae79b919e64\") " Nov 02 23:30:19.471656 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.471503 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a900285e-ba2c-4da2-906b-b8c8f586441c-kube-api-access-2p77f\" (UniqueName: \"kubernetes.io/projected/a900285e-ba2c-4da2-906b-b8c8f586441c-kube-api-access-2p77f\") pod \"a900285e-ba2c-4da2-906b-b8c8f586441c\" (UID: \"a900285e-ba2c-4da2-906b-b8c8f586441c\") " Nov 02 23:30:19.471656 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.471544 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a900285e-ba2c-4da2-906b-b8c8f586441c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a900285e-ba2c-4da2-906b-b8c8f586441c-maas-api-service-ca\") pod \"a900285e-ba2c-4da2-906b-b8c8f586441c\" (UID: \"a900285e-ba2c-4da2-906b-b8c8f586441c\") " Nov 02 23:30:19.471656 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.471627 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-maas-api-service-ca\") pod \"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f\" (UID: \"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f\") " Nov 02 23:30:19.471819 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.471738 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7025314-94d4-4994-a385-eae79b919e64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7025314-94d4-4994-a385-eae79b919e64-maas-api-service-ca\") pod \"e7025314-94d4-4994-a385-eae79b919e64\" (UID: \"e7025314-94d4-4994-a385-eae79b919e64\") " Nov 02 23:30:19.471874 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.471827 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-kube-api-access-plqhw\" (UniqueName: \"kubernetes.io/projected/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-kube-api-access-plqhw\") pod \"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f\" (UID: \"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f\") " Nov 02 23:30:19.472254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.472038 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-maas-api-service-ca" pod "8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" (UID: "8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:19.472254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.472110 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7025314-94d4-4994-a385-eae79b919e64-maas-api-service-ca" pod "e7025314-94d4-4994-a385-eae79b919e64" (UID: "e7025314-94d4-4994-a385-eae79b919e64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:19.472411 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.472340 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7025314-94d4-4994-a385-eae79b919e64-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:19.472411 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.472361 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:19.472799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.472772 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a900285e-ba2c-4da2-906b-b8c8f586441c-maas-api-service-ca" pod "a900285e-ba2c-4da2-906b-b8c8f586441c" (UID: "a900285e-ba2c-4da2-906b-b8c8f586441c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:19.473889 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.473867 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a900285e-ba2c-4da2-906b-b8c8f586441c-kube-api-access-2p77f" pod "a900285e-ba2c-4da2-906b-b8c8f586441c" (UID: "a900285e-ba2c-4da2-906b-b8c8f586441c"). InnerVolumeSpecName "kube-api-access-2p77f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:19.474377 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.474333 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7025314-94d4-4994-a385-eae79b919e64-kube-api-access-2ms57" pod "e7025314-94d4-4994-a385-eae79b919e64" (UID: "e7025314-94d4-4994-a385-eae79b919e64"). InnerVolumeSpecName "kube-api-access-2ms57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:19.474459 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.474372 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-kube-api-access-plqhw" pod "8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" (UID: "8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f"). InnerVolumeSpecName "kube-api-access-plqhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:19.573658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.573583 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plqhw\" (UniqueName: \"kubernetes.io/projected/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f-kube-api-access-plqhw\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:19.573658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.573609 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ms57\" (UniqueName: \"kubernetes.io/projected/e7025314-94d4-4994-a385-eae79b919e64-kube-api-access-2ms57\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:19.573658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.573619 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p77f\" (UniqueName: \"kubernetes.io/projected/a900285e-ba2c-4da2-906b-b8c8f586441c-kube-api-access-2p77f\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:19.573658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:19.573628 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a900285e-ba2c-4da2-906b-b8c8f586441c-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.212879 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.212796 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" Nov 02 23:30:20.213376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.212795 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb" event={"ID":"8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f","Type":"ContainerDied","Data":"19d4cba3b5d68cf5ed849e2a7371dc2c4cfb0cc3135ad3d4f3d34af45173f220"} Nov 02 23:30:20.213376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.212963 5672 scope.go:122] "RemoveContainer" containerID="8cd4d5e0e7f44afe77a686a0d32621ea2409e34996e8af569434faef7aa24087" Nov 02 23:30:20.215202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.215143 5672 generic.go:356] "Generic (PLEG): container finished" podID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerID="c603e530a7b300cfb0e434f942053189396f555f7b9444b4f8892696e42a0d18" exitCode=60 Nov 02 23:30:20.215298 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.215173 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" event={"ID":"88fc9bc8-8e69-4a72-887e-c3612ce0055f","Type":"ContainerDied","Data":"c603e530a7b300cfb0e434f942053189396f555f7b9444b4f8892696e42a0d18"} Nov 02 23:30:20.215664 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.215636 5672 scope.go:122] "RemoveContainer" containerID="c603e530a7b300cfb0e434f942053189396f555f7b9444b4f8892696e42a0d18" Nov 02 23:30:20.215927 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:20.215904 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh_odh-ai-gateway-infra(88fc9bc8-8e69-4a72-887e-c3612ce0055f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" Nov 02 23:30:20.218811 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.218794 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" Nov 02 23:30:20.218897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.218794 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g" event={"ID":"a900285e-ba2c-4da2-906b-b8c8f586441c","Type":"ContainerDied","Data":"d6f15236b3aa5bcf63f1a097fccb5c4f987dd405a3bdbe08d0e399c5553832a3"} Nov 02 23:30:20.221435 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.221413 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" event={"ID":"e7025314-94d4-4994-a385-eae79b919e64","Type":"ContainerDied","Data":"1b7b872a1cdd15e5ed985b5bf719f22d7a3caea076868b8232466fec308411b9"} Nov 02 23:30:20.221532 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.221501 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78" Nov 02 23:30:20.228319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.228294 5672 scope.go:122] "RemoveContainer" containerID="04986fd9d8b2212b60e166cf2f8fb97eb48c0bcb89e456d76f8e54a91efae1ae" Nov 02 23:30:20.255669 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.254364 5672 scope.go:122] "RemoveContainer" containerID="4ba15a58616e12046a85cb556d8832d29854f96970e259fc78227622713500e4" Nov 02 23:30:20.268247 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.267967 5672 scope.go:122] "RemoveContainer" containerID="0e72af92a7b91fa91ce01f41aab05ba6af163a769d0cbeee4b0fd250876d273e" Nov 02 23:30:20.478182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.478136 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78"] Nov 02 23:30:20.481831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.481800 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1c14e6-29894370-r9r78"] Nov 02 23:30:20.517855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.517826 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh"] Nov 02 23:30:20.582008 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.581978 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g"] Nov 02 23:30:20.586907 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.586877 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-02cdd9-29894370-qg85g"] Nov 02 23:30:20.623171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.623127 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d"] Nov 02 23:30:20.689061 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.689029 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb"] Nov 02 23:30:20.691605 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.691577 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894370-bfsnb"] Nov 02 23:30:20.714319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.714284 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" path="/var/lib/kubelet/pods/8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f/volumes" Nov 02 23:30:20.714753 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.714735 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" path="/var/lib/kubelet/pods/a900285e-ba2c-4da2-906b-b8c8f586441c/volumes" Nov 02 23:30:20.715175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.715131 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7025314-94d4-4994-a385-eae79b919e64" path="/var/lib/kubelet/pods/e7025314-94d4-4994-a385-eae79b919e64/volumes" Nov 02 23:30:20.735142 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.735076 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n"] Nov 02 23:30:20.768612 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.768584 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" Nov 02 23:30:20.789493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.789469 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-maas-api-service-ca\") pod \"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c\" (UID: \"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c\") " Nov 02 23:30:20.789626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.789538 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-kube-api-access-j2xxw\" (UniqueName: \"kubernetes.io/projected/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-kube-api-access-j2xxw\") pod \"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c\" (UID: \"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c\") " Nov 02 23:30:20.790509 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.790485 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-maas-api-service-ca" pod "0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" (UID: "0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:20.792132 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.792104 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-kube-api-access-j2xxw" pod "0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" (UID: "0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c"). InnerVolumeSpecName "kube-api-access-j2xxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:20.884985 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.884963 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" Nov 02 23:30:20.891461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.891439 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.891575 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.891465 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2xxw\" (UniqueName: \"kubernetes.io/projected/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c-kube-api-access-j2xxw\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.992482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.992406 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc2da10-0859-46c8-a214-e90ce9f44840-kube-api-access-9sxjk\" (UniqueName: \"kubernetes.io/projected/8fc2da10-0859-46c8-a214-e90ce9f44840-kube-api-access-9sxjk\") pod \"8fc2da10-0859-46c8-a214-e90ce9f44840\" (UID: \"8fc2da10-0859-46c8-a214-e90ce9f44840\") " Nov 02 23:30:20.992619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.992548 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc2da10-0859-46c8-a214-e90ce9f44840-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2da10-0859-46c8-a214-e90ce9f44840-maas-api-service-ca\") pod \"8fc2da10-0859-46c8-a214-e90ce9f44840\" (UID: \"8fc2da10-0859-46c8-a214-e90ce9f44840\") " Nov 02 23:30:20.992915 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.992886 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc2da10-0859-46c8-a214-e90ce9f44840-maas-api-service-ca" pod "8fc2da10-0859-46c8-a214-e90ce9f44840" (UID: "8fc2da10-0859-46c8-a214-e90ce9f44840"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:20.993041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.992966 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc2da10-0859-46c8-a214-e90ce9f44840-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:20.994615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:20.994590 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc2da10-0859-46c8-a214-e90ce9f44840-kube-api-access-9sxjk" pod "8fc2da10-0859-46c8-a214-e90ce9f44840" (UID: "8fc2da10-0859-46c8-a214-e90ce9f44840"). InnerVolumeSpecName "kube-api-access-9sxjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:21.094588 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.094554 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sxjk\" (UniqueName: \"kubernetes.io/projected/8fc2da10-0859-46c8-a214-e90ce9f44840-kube-api-access-9sxjk\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:21.231399 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.231369 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" Nov 02 23:30:21.231913 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.231368 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n" event={"ID":"8fc2da10-0859-46c8-a214-e90ce9f44840","Type":"ContainerDied","Data":"b51587f1839749da2f4ea6e5d479002d1a86efd9491c688e9ced9f39df83291b"} Nov 02 23:30:21.231913 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.231552 5672 scope.go:122] "RemoveContainer" containerID="c6d5ab43f9832ed7c30f5c73f3d8ba9fa07fd9aeb0c847445de09471b715659d" Nov 02 23:30:21.236928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.236905 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" event={"ID":"0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c","Type":"ContainerDied","Data":"0ae05e286d90a2d640ae08324d155c6177b1f8587a6bf797f462f372497d7d68"} Nov 02 23:30:21.237031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.236970 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d" Nov 02 23:30:21.243172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.243127 5672 scope.go:122] "RemoveContainer" containerID="1e9c17be5d09d0aa91d3a06719c291218e65117860d6d036eb6c51c0cf61df44" Nov 02 23:30:21.394742 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.394717 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" Nov 02 23:30:21.493700 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.493629 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d"] Nov 02 23:30:21.498436 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.498410 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88fc9bc8-8e69-4a72-887e-c3612ce0055f-kube-api-access-2q848\" (UniqueName: \"kubernetes.io/projected/88fc9bc8-8e69-4a72-887e-c3612ce0055f-kube-api-access-2q848\") pod \"88fc9bc8-8e69-4a72-887e-c3612ce0055f\" (UID: \"88fc9bc8-8e69-4a72-887e-c3612ce0055f\") " Nov 02 23:30:21.498579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.498512 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88fc9bc8-8e69-4a72-887e-c3612ce0055f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fc9bc8-8e69-4a72-887e-c3612ce0055f-maas-api-service-ca\") pod \"88fc9bc8-8e69-4a72-887e-c3612ce0055f\" (UID: \"88fc9bc8-8e69-4a72-887e-c3612ce0055f\") " Nov 02 23:30:21.498699 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.498678 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894370-86q7d"] Nov 02 23:30:21.498824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.498808 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88fc9bc8-8e69-4a72-887e-c3612ce0055f-maas-api-service-ca" pod "88fc9bc8-8e69-4a72-887e-c3612ce0055f" (UID: "88fc9bc8-8e69-4a72-887e-c3612ce0055f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:30:21.500476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.500459 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fc9bc8-8e69-4a72-887e-c3612ce0055f-kube-api-access-2q848" pod "88fc9bc8-8e69-4a72-887e-c3612ce0055f" (UID: "88fc9bc8-8e69-4a72-887e-c3612ce0055f"). InnerVolumeSpecName "kube-api-access-2q848". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:30:21.600282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.600253 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88fc9bc8-8e69-4a72-887e-c3612ce0055f-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:21.600282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.600279 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q848\" (UniqueName: \"kubernetes.io/projected/88fc9bc8-8e69-4a72-887e-c3612ce0055f-kube-api-access-2q848\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:30:21.600778 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.600739 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n"] Nov 02 23:30:21.603564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:21.603539 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894370-2547n"] Nov 02 23:30:22.244806 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:22.244775 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" Nov 02 23:30:22.245368 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:22.245339 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh" event={"ID":"88fc9bc8-8e69-4a72-887e-c3612ce0055f","Type":"ContainerDied","Data":"a7374fd286aa0c826bf2106cde30c22487f73e5f167034ce12a3fc66fbc1219c"} Nov 02 23:30:22.245469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:22.245418 5672 scope.go:122] "RemoveContainer" containerID="c603e530a7b300cfb0e434f942053189396f555f7b9444b4f8892696e42a0d18" Nov 02 23:30:22.397658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:22.397613 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh"] Nov 02 23:30:22.443232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:22.443202 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894370-tlzqh"] Nov 02 23:30:22.713698 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:22.713624 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" path="/var/lib/kubelet/pods/0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c/volumes" Nov 02 23:30:22.714024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:22.714010 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" path="/var/lib/kubelet/pods/88fc9bc8-8e69-4a72-887e-c3612ce0055f/volumes" Nov 02 23:30:22.714406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:22.714391 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" path="/var/lib/kubelet/pods/8fc2da10-0859-46c8-a214-e90ce9f44840/volumes" Nov 02 23:30:26.021453 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:26.021416 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:30:36.151451 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:30:36.151411 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:30:56.020397 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:30:56.020360 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:31:10.657471 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:10.657390 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:31:20.335926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:20.334560 5672 patch_prober.go:29] interesting pod/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": read tcp 10.134.0.2:37476->10.134.0.146:15021: read: connection reset by peer" start-of-body= Nov 02 23:31:20.335926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:20.335057 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": read tcp 10.134.0.2:37476->10.134.0.146:15021: read: connection reset by peer" Nov 02 23:31:20.612138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:20.612046 5672 generic.go:356] "Generic (PLEG): container finished" podID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerID="8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6" exitCode=0 Nov 02 23:31:20.612138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:20.612117 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" event={"ID":"589ff327-0656-4cc4-ae2f-0c5c2c4d0059","Type":"ContainerDied","Data":"8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6"} Nov 02 23:31:20.612550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:20.612536 5672 scope.go:122] "RemoveContainer" containerID="8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6" Nov 02 23:31:20.614048 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:20.614020 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:31:20.614118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:20.614092 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:31:20.614173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:20.614125 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:31:21.622390 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:21.622351 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" event={"ID":"589ff327-0656-4cc4-ae2f-0c5c2c4d0059","Type":"ContainerStarted","Data":"43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab"} Nov 02 23:31:21.646207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:21.646172 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:31:21.655240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:21.653274 5672 patch_prober.go:29] interesting pod/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:31:21.655240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:21.653337 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 23:31:22.647172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:22.631473 5672 patch_prober.go:29] interesting pod/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:31:22.647172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:22.631543 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 23:31:23.649608 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:23.649565 5672 patch_prober.go:29] interesting pod/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:31:23.650137 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:23.649633 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 23:31:24.632552 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:24.632514 5672 patch_prober.go:29] interesting pod/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:31:24.632848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:24.632816 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 23:31:25.651170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:25.643258 5672 patch_prober.go:29] interesting pod/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:31:25.651170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:25.643322 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 23:31:26.021606 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:31:26.021571 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:31:26.632926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:26.632487 5672 patch_prober.go:29] interesting pod/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 23:31:26.632926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:26.632547 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 23:31:27.634913 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:27.634881 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:31:27.635351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:27.635258 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:31:27.635991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:27.635971 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:31:30.007404 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:30.007362 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb"] Nov 02 23:31:35.008173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.008104 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" podUID="f94e7d1e-d9e0-4268-8125-12e0f6e075c9" containerName="payload-pre-processing" containerID="cri-o://f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24" gracePeriod=25 Nov 02 23:31:35.257386 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.257360 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:31:35.407836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.407746 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-kube-api-access-lbj4x\" (UniqueName: \"kubernetes.io/projected/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-kube-api-access-lbj4x\") pod \"f94e7d1e-d9e0-4268-8125-12e0f6e075c9\" (UID: \"f94e7d1e-d9e0-4268-8125-12e0f6e075c9\") " Nov 02 23:31:35.408007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.407836 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-plugins-config-volume\") pod \"f94e7d1e-d9e0-4268-8125-12e0f6e075c9\" (UID: \"f94e7d1e-d9e0-4268-8125-12e0f6e075c9\") " Nov 02 23:31:35.408263 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.408232 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-plugins-config-volume" pod "f94e7d1e-d9e0-4268-8125-12e0f6e075c9" (UID: "f94e7d1e-d9e0-4268-8125-12e0f6e075c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:31:35.409851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.409833 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-kube-api-access-lbj4x" pod "f94e7d1e-d9e0-4268-8125-12e0f6e075c9" (UID: "f94e7d1e-d9e0-4268-8125-12e0f6e075c9"). InnerVolumeSpecName "kube-api-access-lbj4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:31:35.509315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.509280 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbj4x\" (UniqueName: \"kubernetes.io/projected/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-kube-api-access-lbj4x\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:31:35.509315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.509309 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94e7d1e-d9e0-4268-8125-12e0f6e075c9-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:31:35.731853 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.731816 5672 generic.go:356] "Generic (PLEG): container finished" podID="f94e7d1e-d9e0-4268-8125-12e0f6e075c9" containerID="f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24" exitCode=0 Nov 02 23:31:35.732055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.731860 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" Nov 02 23:31:35.732055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.731900 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" event={"ID":"f94e7d1e-d9e0-4268-8125-12e0f6e075c9","Type":"ContainerDied","Data":"f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24"} Nov 02 23:31:35.732055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.731951 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb" event={"ID":"f94e7d1e-d9e0-4268-8125-12e0f6e075c9","Type":"ContainerDied","Data":"b080501c0f5190265f7fb1c4476f3fba03b90d193d314af3937a0dd7c76595b4"} Nov 02 23:31:35.732055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.731975 5672 scope.go:122] "RemoveContainer" containerID="f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24" Nov 02 23:31:35.743815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.743791 5672 scope.go:122] "RemoveContainer" containerID="f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24" Nov 02 23:31:35.744093 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:31:35.744073 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24\": container with ID starting with f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24 not found: ID does not exist" containerID="f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24" Nov 02 23:31:35.744178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.744103 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24"} err="failed to get container status \"f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24\": rpc error: code = NotFound desc = could not find container \"f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24\": container with ID starting with f0b98b13467744af3a95eb8d8639a95589f0731da3817d3429a1e6903f72cf24 not found: ID does not exist" Nov 02 23:31:35.888543 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.888509 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb"] Nov 02 23:31:35.893011 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:35.892983 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3bf62f-9fbd74847-jfvkb"] Nov 02 23:31:36.717590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:36.717554 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94e7d1e-d9e0-4268-8125-12e0f6e075c9" path="/var/lib/kubelet/pods/f94e7d1e-d9e0-4268-8125-12e0f6e075c9/volumes" Nov 02 23:31:42.778697 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:31:42.778659 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:31:56.021372 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:31:56.021336 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:32:05.805600 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:05.805564 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g"] Nov 02 23:32:05.904207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:05.904168 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4"] Nov 02 23:32:06.466308 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.466268 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp"] Nov 02 23:32:06.467037 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467014 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerName="cleanup" Nov 02 23:32:06.467037 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467035 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467051 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467060 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467077 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e7025314-94d4-4994-a385-eae79b919e64" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467084 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7025314-94d4-4994-a385-eae79b919e64" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467094 5672 cpu_manager.go:414] "containerMap: removing container" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467101 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467115 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467122 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467135 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467143 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467171 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467180 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467193 5672 cpu_manager.go:414] "containerMap: removing container" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467199 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467212 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467219 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467228 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467234 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467246 5672 cpu_manager.go:414] "containerMap: removing container" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467253 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467265 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e7025314-94d4-4994-a385-eae79b919e64" containerName="cleanup" Nov 02 23:32:06.467265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467272 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7025314-94d4-4994-a385-eae79b919e64" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467291 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467298 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467307 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467314 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467323 5672 cpu_manager.go:414] "containerMap: removing container" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467330 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467342 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467348 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467360 5672 cpu_manager.go:414] "containerMap: removing container" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467367 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467382 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f94e7d1e-d9e0-4268-8125-12e0f6e075c9" containerName="payload-pre-processing" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467389 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94e7d1e-d9e0-4268-8125-12e0f6e075c9" containerName="payload-pre-processing" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467411 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467418 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467563 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467575 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7025314-94d4-4994-a385-eae79b919e64" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467584 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467593 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467601 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467610 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467622 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467630 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467638 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7025314-94d4-4994-a385-eae79b919e64" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467646 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467654 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467662 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467671 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467679 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467688 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467698 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467708 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a900285e-ba2c-4da2-906b-b8c8f586441c" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467717 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="88fc9bc8-8e69-4a72-887e-c3612ce0055f" containerName="cleanup" Nov 02 23:32:06.468410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.467727 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94e7d1e-d9e0-4268-8125-12e0f6e075c9" containerName="payload-pre-processing" Nov 02 23:32:06.470721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.470699 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.521559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.521530 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-db5d87-openshift-default-dockercfg-7frc8\"" Nov 02 23:32:06.525212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.525182 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp"] Nov 02 23:32:06.559432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.559392 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-istio-token\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.559616 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.559446 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-socket\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.559616 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.559492 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-data\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.559616 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.559519 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70176952-4b2e-4939-98cd-62d60951971d-istio-podinfo\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.559616 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.559590 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-envoy\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.559842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.559674 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdmg\" (UniqueName: \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-kube-api-access-6gdmg\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.559842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.559719 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-certs\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.559842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.559751 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-credential-socket\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.559842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.559799 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70176952-4b2e-4939-98cd-62d60951971d-istiod-ca-cert\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.662921 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.662873 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-istio-token\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.663138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.662933 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-socket\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.663138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.662987 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-data\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.663138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.663014 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70176952-4b2e-4939-98cd-62d60951971d-istio-podinfo\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.663138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.663089 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-envoy\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.663398 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.663196 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdmg\" (UniqueName: \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-kube-api-access-6gdmg\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.663398 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.663244 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-certs\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.663398 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.663278 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-credential-socket\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.663398 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.663329 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70176952-4b2e-4939-98cd-62d60951971d-istiod-ca-cert\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.664201 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.664175 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70176952-4b2e-4939-98cd-62d60951971d-istiod-ca-cert\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.664826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.664794 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-socket\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.665089 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.665064 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-data\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.668535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.668179 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-certs\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.668535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.668491 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-credential-socket\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.669052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.669006 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-envoy\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.669710 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.669688 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70176952-4b2e-4939-98cd-62d60951971d-istio-podinfo\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.718135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.718104 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-istio-token\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.722802 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.722773 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdmg\" (UniqueName: \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-kube-api-access-6gdmg\") pod \"e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:06.786040 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:06.786001 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:07.208395 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:07.208331 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp"] Nov 02 23:32:07.211485 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:32:07.211424 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70176952_4b2e_4939_98cd_62d60951971d.slice/crio-6f1ac12ac3fb37cb67e459c8492dcac60df935f0bfcb4c4402b86a5ebaaa9cd4 WatchSource:0}: Error finding container 6f1ac12ac3fb37cb67e459c8492dcac60df935f0bfcb4c4402b86a5ebaaa9cd4: Status 404 returned error can't find the container with id 6f1ac12ac3fb37cb67e459c8492dcac60df935f0bfcb4c4402b86a5ebaaa9cd4 Nov 02 23:32:07.214222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:07.214189 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:32:07.214302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:07.214250 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:32:07.214302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:07.214280 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:32:07.940688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:07.940650 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" event={"ID":"70176952-4b2e-4939-98cd-62d60951971d","Type":"ContainerStarted","Data":"2a8502de80f8e38977d13a28b58bab2db9deb5ae5dec88ea619d48171ddfd2c9"} Nov 02 23:32:07.941004 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:07.940707 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" event={"ID":"70176952-4b2e-4939-98cd-62d60951971d","Type":"ContainerStarted","Data":"6f1ac12ac3fb37cb67e459c8492dcac60df935f0bfcb4c4402b86a5ebaaa9cd4"} Nov 02 23:32:08.054175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:08.053624 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" podStartSLOduration=2.053603542 podStartE2EDuration="2.053603542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:08.05001938 +0000 UTC m=+2321.457331210" watchObservedRunningTime="2026-11-02 23:32:08.053603542 +0000 UTC m=+2321.460915374" Nov 02 23:32:08.786868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:08.786834 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:08.792705 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:08.792675 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:08.947216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:08.947176 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:08.948204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:08.948183 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:32:10.905376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:10.905332 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" podUID="424b53ac-a9b0-4f2c-ae87-3a34fcb001fc" containerName="payload-pre-processing" containerID="cri-o://f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6" gracePeriod=25 Nov 02 23:32:10.924038 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:10.924004 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" podUID="eec49111-55b2-4725-9582-44670bdc2f43" containerName="payload-processing" containerID="cri-o://8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451" gracePeriod=25 Nov 02 23:32:11.200710 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.200681 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:32:11.224257 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.224233 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:32:11.314254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.314210 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-kube-api-access-rt672\" (UniqueName: \"kubernetes.io/projected/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-kube-api-access-rt672\") pod \"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc\" (UID: \"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc\") " Nov 02 23:32:11.314438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.314260 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-plugins-config-volume\") pod \"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc\" (UID: \"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc\") " Nov 02 23:32:11.314438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.314345 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec49111-55b2-4725-9582-44670bdc2f43-kube-api-access-p5rlh\" (UniqueName: \"kubernetes.io/projected/eec49111-55b2-4725-9582-44670bdc2f43-kube-api-access-p5rlh\") pod \"eec49111-55b2-4725-9582-44670bdc2f43\" (UID: \"eec49111-55b2-4725-9582-44670bdc2f43\") " Nov 02 23:32:11.314438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.314423 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec49111-55b2-4725-9582-44670bdc2f43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec49111-55b2-4725-9582-44670bdc2f43-plugins-config-volume\") pod \"eec49111-55b2-4725-9582-44670bdc2f43\" (UID: \"eec49111-55b2-4725-9582-44670bdc2f43\") " Nov 02 23:32:11.315237 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.315208 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec49111-55b2-4725-9582-44670bdc2f43-plugins-config-volume" pod "eec49111-55b2-4725-9582-44670bdc2f43" (UID: "eec49111-55b2-4725-9582-44670bdc2f43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:11.315576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.315549 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-plugins-config-volume" pod "424b53ac-a9b0-4f2c-ae87-3a34fcb001fc" (UID: "424b53ac-a9b0-4f2c-ae87-3a34fcb001fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:11.318136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.318103 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec49111-55b2-4725-9582-44670bdc2f43-kube-api-access-p5rlh" pod "eec49111-55b2-4725-9582-44670bdc2f43" (UID: "eec49111-55b2-4725-9582-44670bdc2f43"). InnerVolumeSpecName "kube-api-access-p5rlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:11.318256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.318212 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-kube-api-access-rt672" pod "424b53ac-a9b0-4f2c-ae87-3a34fcb001fc" (UID: "424b53ac-a9b0-4f2c-ae87-3a34fcb001fc"). InnerVolumeSpecName "kube-api-access-rt672". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:11.415887 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.415857 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rlh\" (UniqueName: \"kubernetes.io/projected/eec49111-55b2-4725-9582-44670bdc2f43-kube-api-access-p5rlh\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:11.415887 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.415886 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec49111-55b2-4725-9582-44670bdc2f43-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:11.416069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.415898 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt672\" (UniqueName: \"kubernetes.io/projected/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-kube-api-access-rt672\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:11.416069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.415908 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:11.971348 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.971307 5672 generic.go:356] "Generic (PLEG): container finished" podID="eec49111-55b2-4725-9582-44670bdc2f43" containerID="8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451" exitCode=0 Nov 02 23:32:11.971807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.971374 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" Nov 02 23:32:11.971807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.971405 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" event={"ID":"eec49111-55b2-4725-9582-44670bdc2f43","Type":"ContainerDied","Data":"8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451"} Nov 02 23:32:11.971807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.971467 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g" event={"ID":"eec49111-55b2-4725-9582-44670bdc2f43","Type":"ContainerDied","Data":"243babe8dfafd9e12aec695f8d30dfdac588fc5d0bb150b83ccd3f47b9566890"} Nov 02 23:32:11.972043 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.971496 5672 scope.go:122] "RemoveContainer" containerID="8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451" Nov 02 23:32:11.973582 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.973193 5672 generic.go:356] "Generic (PLEG): container finished" podID="424b53ac-a9b0-4f2c-ae87-3a34fcb001fc" containerID="f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6" exitCode=0 Nov 02 23:32:11.973582 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.973346 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" event={"ID":"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc","Type":"ContainerDied","Data":"f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6"} Nov 02 23:32:11.973582 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.973390 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" event={"ID":"424b53ac-a9b0-4f2c-ae87-3a34fcb001fc","Type":"ContainerDied","Data":"1860bb11aeed6cf840f2fafe87c8490cad14d17c09b9bcbbc102947c4d1f27c3"} Nov 02 23:32:11.973582 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.973428 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4" Nov 02 23:32:11.984055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.984028 5672 scope.go:122] "RemoveContainer" containerID="8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451" Nov 02 23:32:11.984664 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:32:11.984626 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451\": container with ID starting with 8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451 not found: ID does not exist" containerID="8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451" Nov 02 23:32:11.984783 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.984679 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451"} err="failed to get container status \"8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451\": rpc error: code = NotFound desc = could not find container \"8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451\": container with ID starting with 8e2059f400ac5c8c4d7c3b6aabf4fd2a31b541665c3f2e2881b5ef085ce6b451 not found: ID does not exist" Nov 02 23:32:11.984783 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.984706 5672 scope.go:122] "RemoveContainer" containerID="f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6" Nov 02 23:32:11.995205 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.995185 5672 scope.go:122] "RemoveContainer" containerID="f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6" Nov 02 23:32:11.995565 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:32:11.995545 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6\": container with ID starting with f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6 not found: ID does not exist" containerID="f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6" Nov 02 23:32:11.995647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:11.995573 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6"} err="failed to get container status \"f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6\": rpc error: code = NotFound desc = could not find container \"f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6\": container with ID starting with f79789dbcb9d47d67165f18aa8d04253527f06f2cd901d89ff0711fd6769f9b6 not found: ID does not exist" Nov 02 23:32:12.131353 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.131315 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4"] Nov 02 23:32:12.141803 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.141763 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a3909-99d584df9-2dqz4"] Nov 02 23:32:12.253704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.253666 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g"] Nov 02 23:32:12.268685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.268636 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-86d4669679-zt27g"] Nov 02 23:32:12.712732 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.712650 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424b53ac-a9b0-4f2c-ae87-3a34fcb001fc" path="/var/lib/kubelet/pods/424b53ac-a9b0-4f2c-ae87-3a34fcb001fc/volumes" Nov 02 23:32:12.713041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.713028 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec49111-55b2-4725-9582-44670bdc2f43" path="/var/lib/kubelet/pods/eec49111-55b2-4725-9582-44670bdc2f43/volumes" Nov 02 23:32:12.909939 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.909907 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q"] Nov 02 23:32:12.910422 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910407 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerName="cleanup" Nov 02 23:32:12.910473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910423 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerName="cleanup" Nov 02 23:32:12.910473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910440 5672 cpu_manager.go:414] "containerMap: removing container" podUID="424b53ac-a9b0-4f2c-ae87-3a34fcb001fc" containerName="payload-pre-processing" Nov 02 23:32:12.910473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910445 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424b53ac-a9b0-4f2c-ae87-3a34fcb001fc" containerName="payload-pre-processing" Nov 02 23:32:12.910473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910460 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e7025314-94d4-4994-a385-eae79b919e64" containerName="cleanup" Nov 02 23:32:12.910473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910465 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7025314-94d4-4994-a385-eae79b919e64" containerName="cleanup" Nov 02 23:32:12.910473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910474 5672 cpu_manager.go:414] "containerMap: removing container" podUID="eec49111-55b2-4725-9582-44670bdc2f43" containerName="payload-processing" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910479 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec49111-55b2-4725-9582-44670bdc2f43" containerName="payload-processing" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910489 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910494 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910505 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910510 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910516 5672 cpu_manager.go:414] "containerMap: removing container" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910522 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910534 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910538 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910622 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67b7984-ede5-4e8f-b266-34cd84c7f2c1" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910630 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="64237b78-b4e0-42c6-b9e6-3f75e7710ff1" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910636 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc2da10-0859-46c8-a214-e90ce9f44840" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910643 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="424b53ac-a9b0-4f2c-ae87-3a34fcb001fc" containerName="payload-pre-processing" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910651 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7025314-94d4-4994-a385-eae79b919e64" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910657 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec49111-55b2-4725-9582-44670bdc2f43" containerName="payload-processing" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910663 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0498fc-b7d3-4b8c-a83f-0d08ed968c8f" containerName="cleanup" Nov 02 23:32:12.910685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.910671 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4d1076-7133-4df6-8b4b-7a61ff4bdb3c" containerName="cleanup" Nov 02 23:32:12.913969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.913950 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:12.965668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:12.965572 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-0fac41-openshift-default-dockercfg-25nhk\"" Nov 02 23:32:13.010594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.010553 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q"] Nov 02 23:32:13.036553 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.036499 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-certs\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.036755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.036578 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-credential-socket\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.036755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.036703 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbcp\" (UniqueName: \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-kube-api-access-sgbcp\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.036755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.036750 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0083ae5e-d949-47ce-8716-fd952f06f490-istio-podinfo\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.036954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.036813 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-envoy\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.036954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.036869 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0083ae5e-d949-47ce-8716-fd952f06f490-istiod-ca-cert\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.036954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.036930 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-data\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.037123 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.037037 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-istio-token\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.037210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.037139 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-socket\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.138595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.138560 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-socket\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.138780 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.138613 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-certs\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.138780 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.138641 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-credential-socket\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.138780 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.138676 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbcp\" (UniqueName: \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-kube-api-access-sgbcp\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.138780 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.138696 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0083ae5e-d949-47ce-8716-fd952f06f490-istio-podinfo\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.138780 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.138714 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-envoy\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.138780 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.138768 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0083ae5e-d949-47ce-8716-fd952f06f490-istiod-ca-cert\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.139133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.138800 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-data\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.139133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.138869 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-istio-token\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.139133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.139002 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-socket\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.139133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.139073 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-credential-socket\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.139424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.139390 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-data\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.139424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.139415 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-certs\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.139781 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.139757 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0083ae5e-d949-47ce-8716-fd952f06f490-istiod-ca-cert\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.141209 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.141186 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-envoy\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.141314 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.141294 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0083ae5e-d949-47ce-8716-fd952f06f490-istio-podinfo\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.192454 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.192414 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-istio-token\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.192714 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.192691 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbcp\" (UniqueName: \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-kube-api-access-sgbcp\") pod \"e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.229568 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.229537 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:13.648249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.648221 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q"] Nov 02 23:32:13.649745 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:32:13.649716 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0083ae5e_d949_47ce_8716_fd952f06f490.slice/crio-e17780ca2f798832b21ba2519f4dd87197be3aa170cefcd53eda3943c5ba4135 WatchSource:0}: Error finding container e17780ca2f798832b21ba2519f4dd87197be3aa170cefcd53eda3943c5ba4135: Status 404 returned error can't find the container with id e17780ca2f798832b21ba2519f4dd87197be3aa170cefcd53eda3943c5ba4135 Nov 02 23:32:13.652451 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.652416 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:32:13.652557 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.652481 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:32:13.652557 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.652510 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:32:13.990769 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.990727 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" event={"ID":"0083ae5e-d949-47ce-8716-fd952f06f490","Type":"ContainerStarted","Data":"478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a"} Nov 02 23:32:13.990976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:13.990792 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" event={"ID":"0083ae5e-d949-47ce-8716-fd952f06f490","Type":"ContainerStarted","Data":"e17780ca2f798832b21ba2519f4dd87197be3aa170cefcd53eda3943c5ba4135"} Nov 02 23:32:14.102796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:14.102742 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" podStartSLOduration=2.102727329 podStartE2EDuration="2.102727329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:14.099978797 +0000 UTC m=+2327.507290626" watchObservedRunningTime="2026-11-02 23:32:14.102727329 +0000 UTC m=+2327.510039159" Nov 02 23:32:14.230416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:14.230382 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:14.236603 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:14.236578 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:14.997195 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:14.997160 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:14.998212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:14.998187 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:32:26.020029 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:32:26.019990 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:32:41.837401 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:41.837360 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:32:44.001177 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.001127 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns"] Nov 02 23:32:44.006636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.006614 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:44.058022 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.057990 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns"] Nov 02 23:32:44.081585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.081563 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:32:44.081866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.081848 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:32:44.081942 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.081909 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:32:44.082000 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.081938 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:32:44.106701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.106669 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4tnp\" (UniqueName: \"kubernetes.io/projected/198bb295-f72a-421b-951d-17e09725d8aa-kube-api-access-j4tnp\") pod \"maas-api-revoke-sub-e8bd3cd6-8v2ns\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:44.106846 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.106729 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8bd3cd6-8v2ns\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:44.106896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.106875 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8bd3cd6-8v2ns\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:44.208047 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.208014 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4tnp\" (UniqueName: \"kubernetes.io/projected/198bb295-f72a-421b-951d-17e09725d8aa-kube-api-access-j4tnp\") pod \"maas-api-revoke-sub-e8bd3cd6-8v2ns\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:44.208236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.208058 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8bd3cd6-8v2ns\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:44.208236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.208133 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8bd3cd6-8v2ns\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:44.208856 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.208836 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8bd3cd6-8v2ns\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:44.210487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.210465 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8bd3cd6-8v2ns\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:44.261575 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.261509 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4tnp\" (UniqueName: \"kubernetes.io/projected/198bb295-f72a-421b-951d-17e09725d8aa-kube-api-access-j4tnp\") pod \"maas-api-revoke-sub-e8bd3cd6-8v2ns\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:44.320832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.320798 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:44.692160 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:44.692117 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns"] Nov 02 23:32:44.694327 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:32:44.694296 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198bb295_f72a_421b_951d_17e09725d8aa.slice/crio-d99f560964649fd86e8fb5a2ab4f1959db0925c08a03616410f9ae3638923730 WatchSource:0}: Error finding container d99f560964649fd86e8fb5a2ab4f1959db0925c08a03616410f9ae3638923730: Status 404 returned error can't find the container with id d99f560964649fd86e8fb5a2ab4f1959db0925c08a03616410f9ae3638923730 Nov 02 23:32:45.211853 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:45.211816 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" event={"ID":"198bb295-f72a-421b-951d-17e09725d8aa","Type":"ContainerStarted","Data":"fbcdc69b27ae2e6481873fb818da549acb50ba35ea3a5fc4b4087ff41557ab57"} Nov 02 23:32:45.212268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:45.211874 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" event={"ID":"198bb295-f72a-421b-951d-17e09725d8aa","Type":"ContainerStarted","Data":"d99f560964649fd86e8fb5a2ab4f1959db0925c08a03616410f9ae3638923730"} Nov 02 23:32:45.319001 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:45.318904 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" podStartSLOduration=2.318889681 podStartE2EDuration="2.318889681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:45.31683327 +0000 UTC m=+2358.724145102" watchObservedRunningTime="2026-11-02 23:32:45.318889681 +0000 UTC m=+2358.726201510" Nov 02 23:32:46.219961 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:46.219927 5672 generic.go:356] "Generic (PLEG): container finished" podID="198bb295-f72a-421b-951d-17e09725d8aa" containerID="fbcdc69b27ae2e6481873fb818da549acb50ba35ea3a5fc4b4087ff41557ab57" exitCode=0 Nov 02 23:32:46.220382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:46.220013 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" event={"ID":"198bb295-f72a-421b-951d-17e09725d8aa","Type":"ContainerDied","Data":"fbcdc69b27ae2e6481873fb818da549acb50ba35ea3a5fc4b4087ff41557ab57"} Nov 02 23:32:47.373428 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:47.373398 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:47.444670 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:47.444630 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-xks-ca\") pod \"198bb295-f72a-421b-951d-17e09725d8aa\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " Nov 02 23:32:47.444864 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:47.444795 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-openshift-ca\") pod \"198bb295-f72a-421b-951d-17e09725d8aa\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " Nov 02 23:32:47.444935 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:47.444865 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198bb295-f72a-421b-951d-17e09725d8aa-kube-api-access-j4tnp\" (UniqueName: \"kubernetes.io/projected/198bb295-f72a-421b-951d-17e09725d8aa-kube-api-access-j4tnp\") pod \"198bb295-f72a-421b-951d-17e09725d8aa\" (UID: \"198bb295-f72a-421b-951d-17e09725d8aa\") " Nov 02 23:32:47.445223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:47.445196 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-openshift-ca" pod "198bb295-f72a-421b-951d-17e09725d8aa" (UID: "198bb295-f72a-421b-951d-17e09725d8aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:32:47.446861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:47.446837 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-xks-ca" pod "198bb295-f72a-421b-951d-17e09725d8aa" (UID: "198bb295-f72a-421b-951d-17e09725d8aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:32:47.446983 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:47.446967 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198bb295-f72a-421b-951d-17e09725d8aa-kube-api-access-j4tnp" pod "198bb295-f72a-421b-951d-17e09725d8aa" (UID: "198bb295-f72a-421b-951d-17e09725d8aa"). InnerVolumeSpecName "kube-api-access-j4tnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:32:47.546679 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:47.546591 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:47.546679 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:47.546625 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4tnp\" (UniqueName: \"kubernetes.io/projected/198bb295-f72a-421b-951d-17e09725d8aa-kube-api-access-j4tnp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:47.546679 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:47.546639 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/198bb295-f72a-421b-951d-17e09725d8aa-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:32:48.233174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:48.233120 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" Nov 02 23:32:48.233374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:48.233222 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns" event={"ID":"198bb295-f72a-421b-951d-17e09725d8aa","Type":"ContainerDied","Data":"d99f560964649fd86e8fb5a2ab4f1959db0925c08a03616410f9ae3638923730"} Nov 02 23:32:48.233374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:32:48.233271 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d99f560964649fd86e8fb5a2ab4f1959db0925c08a03616410f9ae3638923730" Nov 02 23:32:56.020638 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:32:56.020594 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:33:02.066410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.066371 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz"] Nov 02 23:33:02.067099 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.067076 5672 cpu_manager.go:414] "containerMap: removing container" podUID="198bb295-f72a-421b-951d-17e09725d8aa" containerName="revoke-keys" Nov 02 23:33:02.067204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.067100 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198bb295-f72a-421b-951d-17e09725d8aa" containerName="revoke-keys" Nov 02 23:33:02.067303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.067289 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="198bb295-f72a-421b-951d-17e09725d8aa" containerName="revoke-keys" Nov 02 23:33:02.071024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.070997 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:02.121445 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.121415 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz"] Nov 02 23:33:02.165521 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.165490 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:33:02.165521 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.165511 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:33:02.165723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.165538 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:33:02.166241 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.166222 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:33:02.211844 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.211802 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1387e85c-b6tjz\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:02.212022 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.211991 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ccm\" (UniqueName: \"kubernetes.io/projected/279470ef-32ae-4787-8580-2e1d095204f0-kube-api-access-x7ccm\") pod \"maas-api-revoke-sub-1387e85c-b6tjz\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:02.212097 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.212023 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1387e85c-b6tjz\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:02.312901 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.312862 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1387e85c-b6tjz\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:02.313068 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.312975 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ccm\" (UniqueName: \"kubernetes.io/projected/279470ef-32ae-4787-8580-2e1d095204f0-kube-api-access-x7ccm\") pod \"maas-api-revoke-sub-1387e85c-b6tjz\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:02.313068 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.312994 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1387e85c-b6tjz\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:02.313661 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.313638 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1387e85c-b6tjz\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:02.315213 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.315189 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1387e85c-b6tjz\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:02.366661 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.366602 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ccm\" (UniqueName: \"kubernetes.io/projected/279470ef-32ae-4787-8580-2e1d095204f0-kube-api-access-x7ccm\") pod \"maas-api-revoke-sub-1387e85c-b6tjz\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:02.384276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.384252 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:02.763507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:02.761454 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz"] Nov 02 23:33:03.331554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:03.331460 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" event={"ID":"279470ef-32ae-4787-8580-2e1d095204f0","Type":"ContainerStarted","Data":"ab15b9aa251555289d799d5111a1871c4967fafa1dedc9b1c549ea70ef577a2a"} Nov 02 23:33:03.331554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:03.331521 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" event={"ID":"279470ef-32ae-4787-8580-2e1d095204f0","Type":"ContainerStarted","Data":"44200412598b95e1fa409406e16e8a72fe9e9dfdfb220621fa3d26e8993c6e5e"} Nov 02 23:33:03.438679 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:03.438614 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" podStartSLOduration=1.4385978480000001 podStartE2EDuration="1.438597848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:03.437410046 +0000 UTC m=+2376.844721876" watchObservedRunningTime="2026-11-02 23:33:03.438597848 +0000 UTC m=+2376.845909678" Nov 02 23:33:04.339300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:04.339254 5672 generic.go:356] "Generic (PLEG): container finished" podID="279470ef-32ae-4787-8580-2e1d095204f0" containerID="ab15b9aa251555289d799d5111a1871c4967fafa1dedc9b1c549ea70ef577a2a" exitCode=0 Nov 02 23:33:04.339786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:04.339338 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" event={"ID":"279470ef-32ae-4787-8580-2e1d095204f0","Type":"ContainerDied","Data":"ab15b9aa251555289d799d5111a1871c4967fafa1dedc9b1c549ea70ef577a2a"} Nov 02 23:33:05.493387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:05.493364 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:05.655360 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:05.655272 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279470ef-32ae-4787-8580-2e1d095204f0-kube-api-access-x7ccm\" (UniqueName: \"kubernetes.io/projected/279470ef-32ae-4787-8580-2e1d095204f0-kube-api-access-x7ccm\") pod \"279470ef-32ae-4787-8580-2e1d095204f0\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " Nov 02 23:33:05.655527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:05.655406 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-xks-ca\") pod \"279470ef-32ae-4787-8580-2e1d095204f0\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " Nov 02 23:33:05.655527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:05.655480 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-openshift-ca\") pod \"279470ef-32ae-4787-8580-2e1d095204f0\" (UID: \"279470ef-32ae-4787-8580-2e1d095204f0\") " Nov 02 23:33:05.655896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:05.655861 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-openshift-ca" pod "279470ef-32ae-4787-8580-2e1d095204f0" (UID: "279470ef-32ae-4787-8580-2e1d095204f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:33:05.656025 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:05.656004 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:05.657525 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:05.657493 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-xks-ca" pod "279470ef-32ae-4787-8580-2e1d095204f0" (UID: "279470ef-32ae-4787-8580-2e1d095204f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:33:05.657652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:05.657548 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279470ef-32ae-4787-8580-2e1d095204f0-kube-api-access-x7ccm" pod "279470ef-32ae-4787-8580-2e1d095204f0" (UID: "279470ef-32ae-4787-8580-2e1d095204f0"). InnerVolumeSpecName "kube-api-access-x7ccm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:33:05.756556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:05.756521 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ccm\" (UniqueName: \"kubernetes.io/projected/279470ef-32ae-4787-8580-2e1d095204f0-kube-api-access-x7ccm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:05.756556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:05.756548 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279470ef-32ae-4787-8580-2e1d095204f0-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:06.353572 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:06.353539 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" Nov 02 23:33:06.353572 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:06.353559 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz" event={"ID":"279470ef-32ae-4787-8580-2e1d095204f0","Type":"ContainerDied","Data":"44200412598b95e1fa409406e16e8a72fe9e9dfdfb220621fa3d26e8993c6e5e"} Nov 02 23:33:06.353842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:06.353605 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44200412598b95e1fa409406e16e8a72fe9e9dfdfb220621fa3d26e8993c6e5e" Nov 02 23:33:18.340133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.340096 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x"] Nov 02 23:33:18.340768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.340585 5672 cpu_manager.go:414] "containerMap: removing container" podUID="279470ef-32ae-4787-8580-2e1d095204f0" containerName="revoke-keys" Nov 02 23:33:18.340768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.340598 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279470ef-32ae-4787-8580-2e1d095204f0" containerName="revoke-keys" Nov 02 23:33:18.340768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.340693 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="279470ef-32ae-4787-8580-2e1d095204f0" containerName="revoke-keys" Nov 02 23:33:18.344370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.344345 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:18.396024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.395986 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x"] Nov 02 23:33:18.409649 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.409617 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmrc7\" (UniqueName: \"kubernetes.io/projected/cd134bca-8f4a-4b50-93f6-b6f7b9565877-kube-api-access-nmrc7\") pod \"maas-api-revoke-sub-da1a657c-n9k6x\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:18.409819 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.409681 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da1a657c-n9k6x\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:18.409863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.409843 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da1a657c-n9k6x\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:18.439236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.439207 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:33:18.439424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.439353 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:33:18.439424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.439354 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:33:18.439424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.439408 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:33:18.511443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.511412 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmrc7\" (UniqueName: \"kubernetes.io/projected/cd134bca-8f4a-4b50-93f6-b6f7b9565877-kube-api-access-nmrc7\") pod \"maas-api-revoke-sub-da1a657c-n9k6x\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:18.511634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.511457 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da1a657c-n9k6x\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:18.511634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.511522 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da1a657c-n9k6x\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:18.512230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.512203 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da1a657c-n9k6x\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:18.513787 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.513760 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da1a657c-n9k6x\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:18.567595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.567563 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmrc7\" (UniqueName: \"kubernetes.io/projected/cd134bca-8f4a-4b50-93f6-b6f7b9565877-kube-api-access-nmrc7\") pod \"maas-api-revoke-sub-da1a657c-n9k6x\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:18.660998 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:18.660923 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:19.030445 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:19.030380 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x"] Nov 02 23:33:19.032420 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:33:19.032388 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd134bca_8f4a_4b50_93f6_b6f7b9565877.slice/crio-27167805aa08a909e3a674b49f2b3acfe2a75a3efdcb16a4f75260abe18d8bb8 WatchSource:0}: Error finding container 27167805aa08a909e3a674b49f2b3acfe2a75a3efdcb16a4f75260abe18d8bb8: Status 404 returned error can't find the container with id 27167805aa08a909e3a674b49f2b3acfe2a75a3efdcb16a4f75260abe18d8bb8 Nov 02 23:33:19.442824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:19.442785 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" event={"ID":"cd134bca-8f4a-4b50-93f6-b6f7b9565877","Type":"ContainerStarted","Data":"4f06956f87d483fe25c36adf5b177d38b47aef87398cb4b1c66ac9534a5e2e3f"} Nov 02 23:33:19.443303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:19.442851 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" event={"ID":"cd134bca-8f4a-4b50-93f6-b6f7b9565877","Type":"ContainerStarted","Data":"27167805aa08a909e3a674b49f2b3acfe2a75a3efdcb16a4f75260abe18d8bb8"} Nov 02 23:33:19.550363 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:19.550307 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" podStartSLOduration=1.550292721 podStartE2EDuration="1.550292721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:19.547976482 +0000 UTC m=+2392.955288312" watchObservedRunningTime="2026-11-02 23:33:19.550292721 +0000 UTC m=+2392.957604606" Nov 02 23:33:20.450398 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:20.450310 5672 generic.go:356] "Generic (PLEG): container finished" podID="cd134bca-8f4a-4b50-93f6-b6f7b9565877" containerID="4f06956f87d483fe25c36adf5b177d38b47aef87398cb4b1c66ac9534a5e2e3f" exitCode=0 Nov 02 23:33:20.450398 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:20.450356 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" event={"ID":"cd134bca-8f4a-4b50-93f6-b6f7b9565877","Type":"ContainerDied","Data":"4f06956f87d483fe25c36adf5b177d38b47aef87398cb4b1c66ac9534a5e2e3f"} Nov 02 23:33:21.607874 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:21.607850 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:21.747528 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:21.747493 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd134bca-8f4a-4b50-93f6-b6f7b9565877-kube-api-access-nmrc7\" (UniqueName: \"kubernetes.io/projected/cd134bca-8f4a-4b50-93f6-b6f7b9565877-kube-api-access-nmrc7\") pod \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " Nov 02 23:33:21.747743 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:21.747555 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-openshift-ca\") pod \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " Nov 02 23:33:21.747817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:21.747796 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-xks-ca\") pod \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\" (UID: \"cd134bca-8f4a-4b50-93f6-b6f7b9565877\") " Nov 02 23:33:21.748024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:21.747993 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-openshift-ca" pod "cd134bca-8f4a-4b50-93f6-b6f7b9565877" (UID: "cd134bca-8f4a-4b50-93f6-b6f7b9565877"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:33:21.748312 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:21.748289 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:21.750346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:21.750308 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd134bca-8f4a-4b50-93f6-b6f7b9565877-kube-api-access-nmrc7" pod "cd134bca-8f4a-4b50-93f6-b6f7b9565877" (UID: "cd134bca-8f4a-4b50-93f6-b6f7b9565877"). InnerVolumeSpecName "kube-api-access-nmrc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:33:21.750517 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:21.750491 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-xks-ca" pod "cd134bca-8f4a-4b50-93f6-b6f7b9565877" (UID: "cd134bca-8f4a-4b50-93f6-b6f7b9565877"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:33:21.849317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:21.849237 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd134bca-8f4a-4b50-93f6-b6f7b9565877-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:21.849317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:21.849271 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmrc7\" (UniqueName: \"kubernetes.io/projected/cd134bca-8f4a-4b50-93f6-b6f7b9565877-kube-api-access-nmrc7\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:22.465653 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:22.465609 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" event={"ID":"cd134bca-8f4a-4b50-93f6-b6f7b9565877","Type":"ContainerDied","Data":"27167805aa08a909e3a674b49f2b3acfe2a75a3efdcb16a4f75260abe18d8bb8"} Nov 02 23:33:22.465837 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:22.465695 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27167805aa08a909e3a674b49f2b3acfe2a75a3efdcb16a4f75260abe18d8bb8" Nov 02 23:33:22.465837 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:22.465626 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x" Nov 02 23:33:26.023761 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:33:26.023722 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:33:26.941119 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:26.941087 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:33:27.069616 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:27.069584 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh"] Nov 02 23:33:27.069993 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:27.069776 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerName="istio-proxy" containerID="cri-o://37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438" gracePeriod=30 Nov 02 23:33:32.347899 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.347826 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:33:32.476892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.476853 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-data\") pod \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " Nov 02 23:33:32.477062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.476921 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-credential-socket\") pod \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " Nov 02 23:33:32.477062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.476960 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-token\" (UniqueName: \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-token\") pod \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " Nov 02 23:33:32.477062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.476987 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-podinfo\") pod \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " Nov 02 23:33:32.477062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477031 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istiod-ca-cert\") pod \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " Nov 02 23:33:32.477248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477082 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-envoy\") pod \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " Nov 02 23:33:32.477248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477114 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-socket\") pod \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " Nov 02 23:33:32.477248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477176 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-certs\") pod \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " Nov 02 23:33:32.477248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477171 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-credential-socket" pod "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" (UID: "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:33:32.477248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477170 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-data" pod "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" (UID: "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:33:32.477248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477223 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-kube-api-access-tr585\" (UniqueName: \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-kube-api-access-tr585\") pod \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\" (UID: \"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d\") " Nov 02 23:33:32.478070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477491 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istiod-ca-cert" pod "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" (UID: "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:33:32.478070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477500 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-socket" pod "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" (UID: "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:33:32.478070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477706 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:32.478070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477730 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:32.478070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477745 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:32.478070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477759 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:32.478070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.477983 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-certs" pod "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" (UID: "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:33:32.479378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.479354 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-token" pod "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" (UID: "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:33:32.479617 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.479594 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-kube-api-access-tr585" pod "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" (UID: "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d"). InnerVolumeSpecName "kube-api-access-tr585". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:33:32.479743 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.479724 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-podinfo" pod "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" (UID: "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:33:32.479795 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.479733 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-envoy" pod "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" (UID: "a6eccd0b-f56a-4750-bdc8-6241fc80ea4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:33:32.542287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.542252 5672 generic.go:356] "Generic (PLEG): container finished" podID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerID="37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438" exitCode=0 Nov 02 23:33:32.542480 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.542314 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" Nov 02 23:33:32.542480 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.542340 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" event={"ID":"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d","Type":"ContainerDied","Data":"37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438"} Nov 02 23:33:32.542480 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.542388 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh" event={"ID":"a6eccd0b-f56a-4750-bdc8-6241fc80ea4d","Type":"ContainerDied","Data":"9c26278e66594d439f36d23bca2dd83d250bd0b9a1dcb442ef1241ab29b66a17"} Nov 02 23:33:32.542480 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.542409 5672 scope.go:122] "RemoveContainer" containerID="37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438" Nov 02 23:33:32.562595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.557010 5672 scope.go:122] "RemoveContainer" containerID="37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438" Nov 02 23:33:32.562595 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:33:32.557458 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438\": container with ID starting with 37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438 not found: ID does not exist" containerID="37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438" Nov 02 23:33:32.562595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.557484 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438"} err="failed to get container status \"37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438\": rpc error: code = NotFound desc = could not find container \"37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438\": container with ID starting with 37630f3c2f0a53d1fe2832d4004d2a7497c92cd43146c0d8bf4ccd5f72227438 not found: ID does not exist" Nov 02 23:33:32.578760 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.578729 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:32.578760 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.578753 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:32.578923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.578764 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:32.578923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.578778 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:32.578923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.578791 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr585\" (UniqueName: \"kubernetes.io/projected/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d-kube-api-access-tr585\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:32.696854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.696766 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh"] Nov 02 23:33:32.705107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.705076 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vf4wh"] Nov 02 23:33:32.712678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:32.712655 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" path="/var/lib/kubelet/pods/a6eccd0b-f56a-4750-bdc8-6241fc80ea4d/volumes" Nov 02 23:33:34.359319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.359263 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb"] Nov 02 23:33:34.360040 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.360017 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerName="istio-proxy" Nov 02 23:33:34.360124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.360041 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerName="istio-proxy" Nov 02 23:33:34.360124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.360069 5672 cpu_manager.go:414] "containerMap: removing container" podUID="cd134bca-8f4a-4b50-93f6-b6f7b9565877" containerName="revoke-keys" Nov 02 23:33:34.360124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.360078 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd134bca-8f4a-4b50-93f6-b6f7b9565877" containerName="revoke-keys" Nov 02 23:33:34.360319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.360216 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eccd0b-f56a-4750-bdc8-6241fc80ea4d" containerName="istio-proxy" Nov 02 23:33:34.360319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.360231 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd134bca-8f4a-4b50-93f6-b6f7b9565877" containerName="revoke-keys" Nov 02 23:33:34.365514 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.365485 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.398284 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.398250 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.398441 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.398293 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.398441 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.398331 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.398584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.398484 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmpn\" (UniqueName: \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-kube-api-access-5tmpn\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.398644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.398590 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.398644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.398624 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.398753 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.398716 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.398753 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.398744 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939a37d5-5089-415f-be4c-f5ba19b0de18-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.398873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.398840 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.414317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.414278 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ba1918-openshift-default-dockercfg-k6bc8\"" Nov 02 23:33:34.464984 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.464949 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb"] Nov 02 23:33:34.499553 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.499522 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmpn\" (UniqueName: \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-kube-api-access-5tmpn\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.499739 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.499585 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.499739 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.499607 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.499739 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.499640 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.499739 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.499656 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939a37d5-5089-415f-be4c-f5ba19b0de18-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.499739 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.499711 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.500028 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.499780 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.500028 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.499819 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.500028 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.499861 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.500190 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.500089 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.500240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.500185 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.500240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.500213 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.500476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.500443 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.500625 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.500604 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939a37d5-5089-415f-be4c-f5ba19b0de18-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.502193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.502164 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.502307 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.502277 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.553291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.553257 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.553575 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.553548 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmpn\" (UniqueName: \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-kube-api-access-5tmpn\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:34.678814 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:34.678721 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:35.091935 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.091887 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8"] Nov 02 23:33:35.096135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.096114 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:35.107578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.107553 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6fb8e42-hw6h8\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:35.107686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.107590 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mmv\" (UniqueName: \"kubernetes.io/projected/26fbdc94-3d1e-460a-8a39-188bc40b81a9-kube-api-access-p7mmv\") pod \"maas-api-revoke-sub-e6fb8e42-hw6h8\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:35.107686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.107618 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6fb8e42-hw6h8\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:35.110100 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.110081 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb"] Nov 02 23:33:35.111628 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:33:35.111607 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939a37d5_5089_415f_be4c_f5ba19b0de18.slice/crio-c39fbc9b73392848ab1b023641a3d560dfd0bb1d3c81dbf1184a3dba820b8cd9 WatchSource:0}: Error finding container c39fbc9b73392848ab1b023641a3d560dfd0bb1d3c81dbf1184a3dba820b8cd9: Status 404 returned error can't find the container with id c39fbc9b73392848ab1b023641a3d560dfd0bb1d3c81dbf1184a3dba820b8cd9 Nov 02 23:33:35.114108 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.114080 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:33:35.114194 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.114139 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:33:35.114235 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.114197 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:33:35.149216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.149183 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8"] Nov 02 23:33:35.153618 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.153593 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:33:35.153732 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.153690 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:33:35.153732 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.153730 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:33:35.153851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.153816 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:33:35.208615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.208583 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6fb8e42-hw6h8\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:35.208766 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.208634 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mmv\" (UniqueName: \"kubernetes.io/projected/26fbdc94-3d1e-460a-8a39-188bc40b81a9-kube-api-access-p7mmv\") pod \"maas-api-revoke-sub-e6fb8e42-hw6h8\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:35.208766 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.208689 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6fb8e42-hw6h8\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:35.209259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.209233 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6fb8e42-hw6h8\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:35.210827 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.210806 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6fb8e42-hw6h8\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:35.262126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.262093 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mmv\" (UniqueName: \"kubernetes.io/projected/26fbdc94-3d1e-460a-8a39-188bc40b81a9-kube-api-access-p7mmv\") pod \"maas-api-revoke-sub-e6fb8e42-hw6h8\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:35.410617 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.410534 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:35.578204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.578134 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" event={"ID":"939a37d5-5089-415f-be4c-f5ba19b0de18","Type":"ContainerStarted","Data":"ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7"} Nov 02 23:33:35.578388 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.578227 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" event={"ID":"939a37d5-5089-415f-be4c-f5ba19b0de18","Type":"ContainerStarted","Data":"c39fbc9b73392848ab1b023641a3d560dfd0bb1d3c81dbf1184a3dba820b8cd9"} Nov 02 23:33:35.682320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.682240 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:35.689695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.689630 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" podStartSLOduration=1.6896111729999999 podStartE2EDuration="1.689611173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:35.68753135 +0000 UTC m=+2409.094843201" watchObservedRunningTime="2026-11-02 23:33:35.689611173 +0000 UTC m=+2409.096923004" Nov 02 23:33:35.783423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:35.783391 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8"] Nov 02 23:33:35.785268 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:33:35.785235 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fbdc94_3d1e_460a_8a39_188bc40b81a9.slice/crio-19778aa81bd8644270052c90faeeac3f4a9eacfacdbceb513b79b67ee4626d1a WatchSource:0}: Error finding container 19778aa81bd8644270052c90faeeac3f4a9eacfacdbceb513b79b67ee4626d1a: Status 404 returned error can't find the container with id 19778aa81bd8644270052c90faeeac3f4a9eacfacdbceb513b79b67ee4626d1a Nov 02 23:33:36.587550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:36.587514 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" event={"ID":"26fbdc94-3d1e-460a-8a39-188bc40b81a9","Type":"ContainerStarted","Data":"be64fddd05233486afa0faf5fc89dc5be0fb1fd0b0ccd482241383ab4518f0ea"} Nov 02 23:33:36.587978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:36.587576 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" event={"ID":"26fbdc94-3d1e-460a-8a39-188bc40b81a9","Type":"ContainerStarted","Data":"19778aa81bd8644270052c90faeeac3f4a9eacfacdbceb513b79b67ee4626d1a"} Nov 02 23:33:36.681479 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:36.681442 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 02 23:33:36.681630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:36.681495 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 02 23:33:36.701671 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:36.701625 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" podStartSLOduration=1.7016125290000002 podStartE2EDuration="1.701612529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:36.700072233 +0000 UTC m=+2410.107384066" watchObservedRunningTime="2026-11-02 23:33:36.701612529 +0000 UTC m=+2410.108924358" Nov 02 23:33:37.603524 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:37.603488 5672 generic.go:356] "Generic (PLEG): container finished" podID="26fbdc94-3d1e-460a-8a39-188bc40b81a9" containerID="be64fddd05233486afa0faf5fc89dc5be0fb1fd0b0ccd482241383ab4518f0ea" exitCode=0 Nov 02 23:33:37.604564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:37.604533 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" event={"ID":"26fbdc94-3d1e-460a-8a39-188bc40b81a9","Type":"ContainerDied","Data":"be64fddd05233486afa0faf5fc89dc5be0fb1fd0b0ccd482241383ab4518f0ea"} Nov 02 23:33:37.679243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:37.679209 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 02 23:33:37.679405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:37.679260 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 02 23:33:38.679485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.679444 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 02 23:33:38.679876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.679510 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 02 23:33:38.823003 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.822978 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:38.869074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.868984 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-openshift-ca\") pod \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " Nov 02 23:33:38.869074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.869021 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-xks-ca\") pod \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " Nov 02 23:33:38.869310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.869097 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fbdc94-3d1e-460a-8a39-188bc40b81a9-kube-api-access-p7mmv\" (UniqueName: \"kubernetes.io/projected/26fbdc94-3d1e-460a-8a39-188bc40b81a9-kube-api-access-p7mmv\") pod \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\" (UID: \"26fbdc94-3d1e-460a-8a39-188bc40b81a9\") " Nov 02 23:33:38.869461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.869434 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-openshift-ca" pod "26fbdc94-3d1e-460a-8a39-188bc40b81a9" (UID: "26fbdc94-3d1e-460a-8a39-188bc40b81a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:33:38.871407 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.871372 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-xks-ca" pod "26fbdc94-3d1e-460a-8a39-188bc40b81a9" (UID: "26fbdc94-3d1e-460a-8a39-188bc40b81a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:33:38.872068 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.872044 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fbdc94-3d1e-460a-8a39-188bc40b81a9-kube-api-access-p7mmv" pod "26fbdc94-3d1e-460a-8a39-188bc40b81a9" (UID: "26fbdc94-3d1e-460a-8a39-188bc40b81a9"). InnerVolumeSpecName "kube-api-access-p7mmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:33:38.970791 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.970760 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7mmv\" (UniqueName: \"kubernetes.io/projected/26fbdc94-3d1e-460a-8a39-188bc40b81a9-kube-api-access-p7mmv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:38.970791 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.970788 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:38.970985 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:38.970802 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26fbdc94-3d1e-460a-8a39-188bc40b81a9-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:39.619065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:39.619034 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" Nov 02 23:33:39.619265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:39.619070 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8" event={"ID":"26fbdc94-3d1e-460a-8a39-188bc40b81a9","Type":"ContainerDied","Data":"19778aa81bd8644270052c90faeeac3f4a9eacfacdbceb513b79b67ee4626d1a"} Nov 02 23:33:39.619265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:39.619118 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19778aa81bd8644270052c90faeeac3f4a9eacfacdbceb513b79b67ee4626d1a" Nov 02 23:33:39.679558 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:39.679516 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 02 23:33:39.679997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:39.679576 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 02 23:33:40.679601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:40.679568 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 02 23:33:40.679963 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:40.679616 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 02 23:33:41.679787 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:41.679751 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 02 23:33:41.680174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:41.679807 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 02 23:33:42.683957 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:42.683918 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:42.684504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:42.684483 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:42.685198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:42.685177 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:33:45.838785 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.838746 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm"] Nov 02 23:33:45.839287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.839270 5672 cpu_manager.go:414] "containerMap: removing container" podUID="26fbdc94-3d1e-460a-8a39-188bc40b81a9" containerName="revoke-keys" Nov 02 23:33:45.839332 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.839289 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fbdc94-3d1e-460a-8a39-188bc40b81a9" containerName="revoke-keys" Nov 02 23:33:45.839387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.839377 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fbdc94-3d1e-460a-8a39-188bc40b81a9" containerName="revoke-keys" Nov 02 23:33:45.842923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.842902 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:45.939126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.939086 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:33:45.939328 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.939266 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:33:45.939328 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.939301 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:33:45.939462 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.939325 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:33:45.941891 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.941869 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm"] Nov 02 23:33:45.951832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.951808 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f3256c5-dw9nm\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:45.951935 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.951844 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfsjm\" (UniqueName: \"kubernetes.io/projected/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-kube-api-access-xfsjm\") pod \"maas-api-revoke-sub-1f3256c5-dw9nm\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:45.951975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:45.951926 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f3256c5-dw9nm\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:46.053216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:46.053175 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f3256c5-dw9nm\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:46.053404 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:46.053224 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsjm\" (UniqueName: \"kubernetes.io/projected/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-kube-api-access-xfsjm\") pod \"maas-api-revoke-sub-1f3256c5-dw9nm\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:46.053404 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:46.053286 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f3256c5-dw9nm\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:46.053950 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:46.053928 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f3256c5-dw9nm\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:46.055526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:46.055507 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f3256c5-dw9nm\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:46.106605 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:46.106539 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsjm\" (UniqueName: \"kubernetes.io/projected/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-kube-api-access-xfsjm\") pod \"maas-api-revoke-sub-1f3256c5-dw9nm\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:46.158746 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:46.158722 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:46.523360 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:46.523337 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm"] Nov 02 23:33:46.525076 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:33:46.525044 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1340a2c0_2f2f_4836_a8bd_2f3d3cac4efc.slice/crio-ba114b4dfa7366127dd020f9f1fa7daad3ff834afd2ceabae03ce0eb75586cd9 WatchSource:0}: Error finding container ba114b4dfa7366127dd020f9f1fa7daad3ff834afd2ceabae03ce0eb75586cd9: Status 404 returned error can't find the container with id ba114b4dfa7366127dd020f9f1fa7daad3ff834afd2ceabae03ce0eb75586cd9 Nov 02 23:33:46.666881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:46.666835 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" event={"ID":"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc","Type":"ContainerStarted","Data":"ba114b4dfa7366127dd020f9f1fa7daad3ff834afd2ceabae03ce0eb75586cd9"} Nov 02 23:33:47.675721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:47.675626 5672 generic.go:356] "Generic (PLEG): container finished" podID="1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc" containerID="a8f85eb9298a95a87560c4a44bfb3eb7a169cd74aa3e6513d9da7017c472dbb6" exitCode=0 Nov 02 23:33:47.676109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:47.675713 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" event={"ID":"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc","Type":"ContainerDied","Data":"a8f85eb9298a95a87560c4a44bfb3eb7a169cd74aa3e6513d9da7017c472dbb6"} Nov 02 23:33:48.851793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:48.851765 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:48.991354 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:48.991324 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-xks-ca\") pod \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " Nov 02 23:33:48.991530 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:48.991394 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-openshift-ca\") pod \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " Nov 02 23:33:48.991530 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:48.991512 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-kube-api-access-xfsjm\" (UniqueName: \"kubernetes.io/projected/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-kube-api-access-xfsjm\") pod \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\" (UID: \"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc\") " Nov 02 23:33:48.992126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:48.992093 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-openshift-ca" pod "1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc" (UID: "1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:33:48.993571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:48.993543 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-xks-ca" pod "1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc" (UID: "1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:33:48.993745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:48.993717 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-kube-api-access-xfsjm" pod "1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc" (UID: "1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc"). InnerVolumeSpecName "kube-api-access-xfsjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:33:49.092882 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:49.092848 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfsjm\" (UniqueName: \"kubernetes.io/projected/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-kube-api-access-xfsjm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:49.092882 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:49.092878 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:49.092882 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:49.092888 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:33:49.698915 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:49.698881 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" Nov 02 23:33:49.699240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:49.698915 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm" event={"ID":"1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc","Type":"ContainerDied","Data":"ba114b4dfa7366127dd020f9f1fa7daad3ff834afd2ceabae03ce0eb75586cd9"} Nov 02 23:33:49.699240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:49.698972 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba114b4dfa7366127dd020f9f1fa7daad3ff834afd2ceabae03ce0eb75586cd9" Nov 02 23:33:55.995531 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:55.995496 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq"] Nov 02 23:33:55.996881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:55.996850 5672 cpu_manager.go:414] "containerMap: removing container" podUID="1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc" containerName="revoke-keys" Nov 02 23:33:55.997025 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:55.996924 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc" containerName="revoke-keys" Nov 02 23:33:55.997116 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:55.997086 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc" containerName="revoke-keys" Nov 02 23:33:56.000494 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.000473 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:33:56.028855 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:33:56.028819 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:33:56.051094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.051067 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq"] Nov 02 23:33:56.093412 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.093379 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:33:56.093412 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.093387 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:33:56.095469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.095443 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:33:56.095469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.095465 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:33:56.098439 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.098281 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch"] Nov 02 23:33:56.171765 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.171731 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2pm\" (UniqueName: \"kubernetes.io/projected/2d66e697-a76a-4847-8635-ef632da18e0b-kube-api-access-qk2pm\") pod \"maas-api-revoke-sub-4c207868-clljq\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:33:56.171922 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.171904 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c207868-clljq\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:33:56.172069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.172053 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c207868-clljq\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:33:56.198668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.198630 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw"] Nov 02 23:33:56.273662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.273575 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c207868-clljq\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:33:56.273662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.273641 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2pm\" (UniqueName: \"kubernetes.io/projected/2d66e697-a76a-4847-8635-ef632da18e0b-kube-api-access-qk2pm\") pod \"maas-api-revoke-sub-4c207868-clljq\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:33:56.273887 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.273758 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c207868-clljq\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:33:56.274388 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.274364 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c207868-clljq\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:33:56.275924 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.275901 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c207868-clljq\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:33:56.328357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.328323 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2pm\" (UniqueName: \"kubernetes.io/projected/2d66e697-a76a-4847-8635-ef632da18e0b-kube-api-access-qk2pm\") pod \"maas-api-revoke-sub-4c207868-clljq\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:33:56.615033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.614944 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:33:56.995654 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:56.995623 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq"] Nov 02 23:33:56.997996 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:33:56.997963 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d66e697_a76a_4847_8635_ef632da18e0b.slice/crio-88bad505d858d71ebc3157c90219a7951861d84f58325f8bda48350bd6da61f4 WatchSource:0}: Error finding container 88bad505d858d71ebc3157c90219a7951861d84f58325f8bda48350bd6da61f4: Status 404 returned error can't find the container with id 88bad505d858d71ebc3157c90219a7951861d84f58325f8bda48350bd6da61f4 Nov 02 23:33:57.761426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:57.761381 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" event={"ID":"2d66e697-a76a-4847-8635-ef632da18e0b","Type":"ContainerStarted","Data":"366e5239c0375ca8e80c0c93bf24205d9758557fa636b5846968ae6c2bee2105"} Nov 02 23:33:57.761609 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:57.761452 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" event={"ID":"2d66e697-a76a-4847-8635-ef632da18e0b","Type":"ContainerStarted","Data":"88bad505d858d71ebc3157c90219a7951861d84f58325f8bda48350bd6da61f4"} Nov 02 23:33:57.870438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:57.870377 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" podStartSLOduration=2.870359661 podStartE2EDuration="2.870359661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:57.866447241 +0000 UTC m=+2431.273759072" watchObservedRunningTime="2026-11-02 23:33:57.870359661 +0000 UTC m=+2431.277671493" Nov 02 23:33:58.771615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:58.771576 5672 generic.go:356] "Generic (PLEG): container finished" podID="2d66e697-a76a-4847-8635-ef632da18e0b" containerID="366e5239c0375ca8e80c0c93bf24205d9758557fa636b5846968ae6c2bee2105" exitCode=0 Nov 02 23:33:58.772081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:33:58.771661 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" event={"ID":"2d66e697-a76a-4847-8635-ef632da18e0b","Type":"ContainerDied","Data":"366e5239c0375ca8e80c0c93bf24205d9758557fa636b5846968ae6c2bee2105"} Nov 02 23:34:00.096952 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.096574 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:34:00.182439 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.181501 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-xks-ca\") pod \"2d66e697-a76a-4847-8635-ef632da18e0b\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " Nov 02 23:34:00.182439 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.181580 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d66e697-a76a-4847-8635-ef632da18e0b-kube-api-access-qk2pm\" (UniqueName: \"kubernetes.io/projected/2d66e697-a76a-4847-8635-ef632da18e0b-kube-api-access-qk2pm\") pod \"2d66e697-a76a-4847-8635-ef632da18e0b\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " Nov 02 23:34:00.182439 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.181787 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-openshift-ca\") pod \"2d66e697-a76a-4847-8635-ef632da18e0b\" (UID: \"2d66e697-a76a-4847-8635-ef632da18e0b\") " Nov 02 23:34:00.182998 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.182942 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-openshift-ca" pod "2d66e697-a76a-4847-8635-ef632da18e0b" (UID: "2d66e697-a76a-4847-8635-ef632da18e0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:34:00.192485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.192433 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d66e697-a76a-4847-8635-ef632da18e0b-kube-api-access-qk2pm" pod "2d66e697-a76a-4847-8635-ef632da18e0b" (UID: "2d66e697-a76a-4847-8635-ef632da18e0b"). InnerVolumeSpecName "kube-api-access-qk2pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:34:00.205121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.205077 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-xks-ca" pod "2d66e697-a76a-4847-8635-ef632da18e0b" (UID: "2d66e697-a76a-4847-8635-ef632da18e0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:34:00.297954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.297911 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:00.298210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.298192 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d66e697-a76a-4847-8635-ef632da18e0b-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:00.298315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.298304 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk2pm\" (UniqueName: \"kubernetes.io/projected/2d66e697-a76a-4847-8635-ef632da18e0b-kube-api-access-qk2pm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:00.828537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.828496 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" event={"ID":"2d66e697-a76a-4847-8635-ef632da18e0b","Type":"ContainerDied","Data":"88bad505d858d71ebc3157c90219a7951861d84f58325f8bda48350bd6da61f4"} Nov 02 23:34:00.828818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.828800 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88bad505d858d71ebc3157c90219a7951861d84f58325f8bda48350bd6da61f4" Nov 02 23:34:00.828985 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:00.828973 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq" Nov 02 23:34:01.098790 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:01.098696 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" podUID="ce8f5119-0c53-41e6-a97c-1984fffa7aa5" containerName="payload-processing" containerID="cri-o://94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c" gracePeriod=25 Nov 02 23:34:01.199550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:01.199509 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" podUID="8d33def5-1207-4710-8abd-04250c8cc5a4" containerName="payload-pre-processing" containerID="cri-o://7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048" gracePeriod=25 Nov 02 23:34:02.166495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.166453 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" podUID="ce8f5119-0c53-41e6-a97c-1984fffa7aa5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:34:02.176320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.176237 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" podUID="8d33def5-1207-4710-8abd-04250c8cc5a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:34:02.381892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.381850 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9"] Nov 02 23:34:02.382900 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.382870 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2d66e697-a76a-4847-8635-ef632da18e0b" containerName="revoke-keys" Nov 02 23:34:02.383071 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.383056 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d66e697-a76a-4847-8635-ef632da18e0b" containerName="revoke-keys" Nov 02 23:34:02.383418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.383398 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d66e697-a76a-4847-8635-ef632da18e0b" containerName="revoke-keys" Nov 02 23:34:02.393224 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.390561 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:02.450886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.450648 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9"] Nov 02 23:34:02.488189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.483273 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:34:02.488189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.483380 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:34:02.499303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.499261 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e9c50cd-gbhc9\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:02.499591 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.499564 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e9c50cd-gbhc9\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:02.499894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.499874 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4p9\" (UniqueName: \"kubernetes.io/projected/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-kube-api-access-zt4p9\") pod \"maas-api-revoke-sub-2e9c50cd-gbhc9\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:02.503229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.502205 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:34:02.503229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.502280 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:34:02.600519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.600482 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4p9\" (UniqueName: \"kubernetes.io/projected/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-kube-api-access-zt4p9\") pod \"maas-api-revoke-sub-2e9c50cd-gbhc9\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:02.600696 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.600571 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e9c50cd-gbhc9\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:02.600696 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.600620 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e9c50cd-gbhc9\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:02.601625 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.601599 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e9c50cd-gbhc9\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:02.603274 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.603247 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e9c50cd-gbhc9\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:02.686990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.686919 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4p9\" (UniqueName: \"kubernetes.io/projected/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-kube-api-access-zt4p9\") pod \"maas-api-revoke-sub-2e9c50cd-gbhc9\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:02.706385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:02.706293 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:03.144176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:03.142919 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9"] Nov 02 23:34:03.906468 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:03.906383 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" event={"ID":"0aed28f6-cf92-44e9-b6bd-5b3297e5208c","Type":"ContainerStarted","Data":"c22135e6eba8bc7a8b6ab1a1607d050a20d93e26c3ddf0938f566f604ed849e7"} Nov 02 23:34:03.906468 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:03.906441 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" event={"ID":"0aed28f6-cf92-44e9-b6bd-5b3297e5208c","Type":"ContainerStarted","Data":"71d1c40b9755a4b5530daec966ab7c764f7d95cf5f6548a81aebb281abd73a46"} Nov 02 23:34:04.016084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:04.016026 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" podStartSLOduration=2.016013531 podStartE2EDuration="2.016013531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:34:04.015275345 +0000 UTC m=+2437.422587175" watchObservedRunningTime="2026-11-02 23:34:04.016013531 +0000 UTC m=+2437.423325360" Nov 02 23:34:04.916529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:04.916485 5672 generic.go:356] "Generic (PLEG): container finished" podID="0aed28f6-cf92-44e9-b6bd-5b3297e5208c" containerID="c22135e6eba8bc7a8b6ab1a1607d050a20d93e26c3ddf0938f566f604ed849e7" exitCode=0 Nov 02 23:34:04.917007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:04.916563 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" event={"ID":"0aed28f6-cf92-44e9-b6bd-5b3297e5208c","Type":"ContainerDied","Data":"c22135e6eba8bc7a8b6ab1a1607d050a20d93e26c3ddf0938f566f604ed849e7"} Nov 02 23:34:06.128595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.128560 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:06.151928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.151894 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-xks-ca\") pod \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " Nov 02 23:34:06.152131 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.152005 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-openshift-ca\") pod \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " Nov 02 23:34:06.152131 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.152090 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-kube-api-access-zt4p9\" (UniqueName: \"kubernetes.io/projected/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-kube-api-access-zt4p9\") pod \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\" (UID: \"0aed28f6-cf92-44e9-b6bd-5b3297e5208c\") " Nov 02 23:34:06.152715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.152661 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-openshift-ca" pod "0aed28f6-cf92-44e9-b6bd-5b3297e5208c" (UID: "0aed28f6-cf92-44e9-b6bd-5b3297e5208c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:34:06.155817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.155773 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-kube-api-access-zt4p9" pod "0aed28f6-cf92-44e9-b6bd-5b3297e5208c" (UID: "0aed28f6-cf92-44e9-b6bd-5b3297e5208c"). InnerVolumeSpecName "kube-api-access-zt4p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:34:06.162223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.162183 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-xks-ca" pod "0aed28f6-cf92-44e9-b6bd-5b3297e5208c" (UID: "0aed28f6-cf92-44e9-b6bd-5b3297e5208c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:34:06.257065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.256979 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:06.257065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.257021 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:06.257065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.257037 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt4p9\" (UniqueName: \"kubernetes.io/projected/0aed28f6-cf92-44e9-b6bd-5b3297e5208c-kube-api-access-zt4p9\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:06.937972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.937913 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" event={"ID":"0aed28f6-cf92-44e9-b6bd-5b3297e5208c","Type":"ContainerDied","Data":"71d1c40b9755a4b5530daec966ab7c764f7d95cf5f6548a81aebb281abd73a46"} Nov 02 23:34:06.938191 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.937994 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71d1c40b9755a4b5530daec966ab7c764f7d95cf5f6548a81aebb281abd73a46" Nov 02 23:34:06.938191 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:06.938066 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9" Nov 02 23:34:07.948623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:07.948524 5672 generic.go:356] "Generic (PLEG): container finished" podID="0083ae5e-d949-47ce-8716-fd952f06f490" containerID="478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a" exitCode=0 Nov 02 23:34:07.949209 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:07.949140 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" event={"ID":"0083ae5e-d949-47ce-8716-fd952f06f490","Type":"ContainerDied","Data":"478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a"} Nov 02 23:34:07.949594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:07.949578 5672 scope.go:122] "RemoveContainer" containerID="478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a" Nov 02 23:34:07.951202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:07.951171 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:34:07.951316 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:07.951240 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:34:07.951316 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:07.951274 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:34:08.951173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:08.950793 5672 patch_prober.go:29] interesting pod/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:34:08.951173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:08.950889 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 23:34:08.958266 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:08.958240 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" event={"ID":"0083ae5e-d949-47ce-8716-fd952f06f490","Type":"ContainerStarted","Data":"ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c"} Nov 02 23:34:08.960009 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:08.959988 5672 generic.go:356] "Generic (PLEG): container finished" podID="70176952-4b2e-4939-98cd-62d60951971d" containerID="2a8502de80f8e38977d13a28b58bab2db9deb5ae5dec88ea619d48171ddfd2c9" exitCode=0 Nov 02 23:34:08.960133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:08.960054 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" event={"ID":"70176952-4b2e-4939-98cd-62d60951971d","Type":"ContainerDied","Data":"2a8502de80f8e38977d13a28b58bab2db9deb5ae5dec88ea619d48171ddfd2c9"} Nov 02 23:34:08.960436 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:08.960419 5672 scope.go:122] "RemoveContainer" containerID="2a8502de80f8e38977d13a28b58bab2db9deb5ae5dec88ea619d48171ddfd2c9" Nov 02 23:34:08.962757 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:08.962728 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:34:08.962841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:08.962794 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:34:08.962841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:08.962826 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:34:09.237170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:09.234303 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:34:09.237170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:09.234446 5672 patch_prober.go:29] interesting pod/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 23:34:09.237170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:09.234526 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 23:34:09.982950 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:09.982910 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" event={"ID":"70176952-4b2e-4939-98cd-62d60951971d","Type":"ContainerStarted","Data":"daed650db1f729e39703a0558839d434d0e7fedce00dae977eabe5e52be73d2b"} Nov 02 23:34:10.234048 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:10.233537 5672 patch_prober.go:29] interesting pod/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 23:34:10.234048 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:10.233600 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 23:34:10.787122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:10.787089 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:34:10.787350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:10.787325 5672 patch_prober.go:29] interesting pod/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:34:10.787408 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:10.787388 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 23:34:11.244171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:11.233483 5672 patch_prober.go:29] interesting pod/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 23:34:11.244171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:11.233538 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 23:34:11.786623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:11.786581 5672 patch_prober.go:29] interesting pod/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:34:11.786840 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:11.786647 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 23:34:12.149184 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:12.149071 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" podUID="8d33def5-1207-4710-8abd-04250c8cc5a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:34:12.153260 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:12.153226 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" podUID="ce8f5119-0c53-41e6-a97c-1984fffa7aa5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 23:34:12.231572 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:12.231536 5672 patch_prober.go:29] interesting pod/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 23:34:12.231855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:12.231826 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 23:34:12.787005 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:12.786963 5672 patch_prober.go:29] interesting pod/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:34:12.787559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:12.787018 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 23:34:13.190902 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.190819 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr"] Nov 02 23:34:13.191332 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.191317 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0aed28f6-cf92-44e9-b6bd-5b3297e5208c" containerName="revoke-keys" Nov 02 23:34:13.191387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.191332 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aed28f6-cf92-44e9-b6bd-5b3297e5208c" containerName="revoke-keys" Nov 02 23:34:13.191442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.191433 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aed28f6-cf92-44e9-b6bd-5b3297e5208c" containerName="revoke-keys" Nov 02 23:34:13.194161 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.194125 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:13.246173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.239515 5672 patch_prober.go:29] interesting pod/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 02 23:34:13.246173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.239575 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 02 23:34:13.286015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.285980 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6777eb6-z56fr\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:13.286222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.286046 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kbbs\" (UniqueName: \"kubernetes.io/projected/938f8589-bbe7-4ef6-8fe8-43c81f550222-kube-api-access-9kbbs\") pod \"maas-api-revoke-sub-c6777eb6-z56fr\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:13.286271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.286228 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6777eb6-z56fr\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:13.293975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.293953 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:34:13.294125 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.293958 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:34:13.294243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.293989 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:34:13.294292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.293998 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:34:13.297343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.297321 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr"] Nov 02 23:34:13.387666 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.387632 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6777eb6-z56fr\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:13.387866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.387775 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6777eb6-z56fr\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:13.387866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.387819 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kbbs\" (UniqueName: \"kubernetes.io/projected/938f8589-bbe7-4ef6-8fe8-43c81f550222-kube-api-access-9kbbs\") pod \"maas-api-revoke-sub-c6777eb6-z56fr\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:13.388509 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.388488 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6777eb6-z56fr\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:13.390053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.390013 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6777eb6-z56fr\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:13.448939 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.448862 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kbbs\" (UniqueName: \"kubernetes.io/projected/938f8589-bbe7-4ef6-8fe8-43c81f550222-kube-api-access-9kbbs\") pod \"maas-api-revoke-sub-c6777eb6-z56fr\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:13.506121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.506085 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:13.793174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.789493 5672 patch_prober.go:29] interesting pod/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:34:13.793174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.789566 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 23:34:13.960546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:13.950539 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr"] Nov 02 23:34:14.010409 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:14.010377 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" event={"ID":"938f8589-bbe7-4ef6-8fe8-43c81f550222","Type":"ContainerStarted","Data":"077bb8a16103275adda990bea66abd54faa1d1e599092e54f1e9fa85fa9b6373"} Nov 02 23:34:14.245507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:14.245475 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:34:14.247354 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:14.247334 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:34:14.252420 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:14.252401 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:34:14.787346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:14.787310 5672 patch_prober.go:29] interesting pod/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 23:34:14.787560 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:14.787372 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 23:34:15.018136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:15.018099 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" event={"ID":"938f8589-bbe7-4ef6-8fe8-43c81f550222","Type":"ContainerStarted","Data":"1ea1dc14260f9285314ab10abb2cd51c23487ab04fe456402f6647569c1ea7f8"} Nov 02 23:34:15.124773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:15.124683 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" podStartSLOduration=2.124669256 podStartE2EDuration="2.124669256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:34:15.123398404 +0000 UTC m=+2448.530710234" watchObservedRunningTime="2026-11-02 23:34:15.124669256 +0000 UTC m=+2448.531981121" Nov 02 23:34:15.791833 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:15.791804 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:34:15.792309 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:15.792288 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:34:15.792880 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:15.792862 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:34:16.025383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:16.025347 5672 generic.go:356] "Generic (PLEG): container finished" podID="938f8589-bbe7-4ef6-8fe8-43c81f550222" containerID="1ea1dc14260f9285314ab10abb2cd51c23487ab04fe456402f6647569c1ea7f8" exitCode=0 Nov 02 23:34:16.025815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:16.025428 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" event={"ID":"938f8589-bbe7-4ef6-8fe8-43c81f550222","Type":"ContainerDied","Data":"1ea1dc14260f9285314ab10abb2cd51c23487ab04fe456402f6647569c1ea7f8"} Nov 02 23:34:17.173902 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:17.173879 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:17.233748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:17.233718 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-xks-ca\") pod \"938f8589-bbe7-4ef6-8fe8-43c81f550222\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " Nov 02 23:34:17.233927 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:17.233762 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-openshift-ca\") pod \"938f8589-bbe7-4ef6-8fe8-43c81f550222\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " Nov 02 23:34:17.233927 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:17.233813 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938f8589-bbe7-4ef6-8fe8-43c81f550222-kube-api-access-9kbbs\" (UniqueName: \"kubernetes.io/projected/938f8589-bbe7-4ef6-8fe8-43c81f550222-kube-api-access-9kbbs\") pod \"938f8589-bbe7-4ef6-8fe8-43c81f550222\" (UID: \"938f8589-bbe7-4ef6-8fe8-43c81f550222\") " Nov 02 23:34:17.234233 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:17.234206 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-openshift-ca" pod "938f8589-bbe7-4ef6-8fe8-43c81f550222" (UID: "938f8589-bbe7-4ef6-8fe8-43c81f550222"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:34:17.235875 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:17.235846 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938f8589-bbe7-4ef6-8fe8-43c81f550222-kube-api-access-9kbbs" pod "938f8589-bbe7-4ef6-8fe8-43c81f550222" (UID: "938f8589-bbe7-4ef6-8fe8-43c81f550222"). InnerVolumeSpecName "kube-api-access-9kbbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:34:17.235967 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:17.235869 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-xks-ca" pod "938f8589-bbe7-4ef6-8fe8-43c81f550222" (UID: "938f8589-bbe7-4ef6-8fe8-43c81f550222"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:34:17.335634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:17.335555 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kbbs\" (UniqueName: \"kubernetes.io/projected/938f8589-bbe7-4ef6-8fe8-43c81f550222-kube-api-access-9kbbs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:17.335634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:17.335583 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:17.335634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:17.335593 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938f8589-bbe7-4ef6-8fe8-43c81f550222-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:18.039822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:18.039795 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" Nov 02 23:34:18.040034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:18.039800 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr" event={"ID":"938f8589-bbe7-4ef6-8fe8-43c81f550222","Type":"ContainerDied","Data":"077bb8a16103275adda990bea66abd54faa1d1e599092e54f1e9fa85fa9b6373"} Nov 02 23:34:18.040034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:18.039907 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="077bb8a16103275adda990bea66abd54faa1d1e599092e54f1e9fa85fa9b6373" Nov 02 23:34:21.359248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.359171 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:34:21.377102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.377075 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-kube-api-access-fjj5l\" (UniqueName: \"kubernetes.io/projected/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-kube-api-access-fjj5l\") pod \"ce8f5119-0c53-41e6-a97c-1984fffa7aa5\" (UID: \"ce8f5119-0c53-41e6-a97c-1984fffa7aa5\") " Nov 02 23:34:21.377292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.377181 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-plugins-config-volume\") pod \"ce8f5119-0c53-41e6-a97c-1984fffa7aa5\" (UID: \"ce8f5119-0c53-41e6-a97c-1984fffa7aa5\") " Nov 02 23:34:21.377678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.377604 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-plugins-config-volume" pod "ce8f5119-0c53-41e6-a97c-1984fffa7aa5" (UID: "ce8f5119-0c53-41e6-a97c-1984fffa7aa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:34:21.379818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.379779 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-kube-api-access-fjj5l" pod "ce8f5119-0c53-41e6-a97c-1984fffa7aa5" (UID: "ce8f5119-0c53-41e6-a97c-1984fffa7aa5"). InnerVolumeSpecName "kube-api-access-fjj5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:34:21.461177 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.461139 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:34:21.478487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.478453 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d33def5-1207-4710-8abd-04250c8cc5a4-kube-api-access-rl4sj\" (UniqueName: \"kubernetes.io/projected/8d33def5-1207-4710-8abd-04250c8cc5a4-kube-api-access-rl4sj\") pod \"8d33def5-1207-4710-8abd-04250c8cc5a4\" (UID: \"8d33def5-1207-4710-8abd-04250c8cc5a4\") " Nov 02 23:34:21.478679 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.478538 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d33def5-1207-4710-8abd-04250c8cc5a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d33def5-1207-4710-8abd-04250c8cc5a4-plugins-config-volume\") pod \"8d33def5-1207-4710-8abd-04250c8cc5a4\" (UID: \"8d33def5-1207-4710-8abd-04250c8cc5a4\") " Nov 02 23:34:21.478977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.478944 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d33def5-1207-4710-8abd-04250c8cc5a4-plugins-config-volume" pod "8d33def5-1207-4710-8abd-04250c8cc5a4" (UID: "8d33def5-1207-4710-8abd-04250c8cc5a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:34:21.479104 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.479035 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:21.479104 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.479052 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d33def5-1207-4710-8abd-04250c8cc5a4-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:21.479104 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.479067 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjj5l\" (UniqueName: \"kubernetes.io/projected/ce8f5119-0c53-41e6-a97c-1984fffa7aa5-kube-api-access-fjj5l\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:21.481264 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.481187 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d33def5-1207-4710-8abd-04250c8cc5a4-kube-api-access-rl4sj" pod "8d33def5-1207-4710-8abd-04250c8cc5a4" (UID: "8d33def5-1207-4710-8abd-04250c8cc5a4"). InnerVolumeSpecName "kube-api-access-rl4sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:34:21.579996 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:21.579964 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl4sj\" (UniqueName: \"kubernetes.io/projected/8d33def5-1207-4710-8abd-04250c8cc5a4-kube-api-access-rl4sj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:22.067619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.067583 5672 generic.go:356] "Generic (PLEG): container finished" podID="ce8f5119-0c53-41e6-a97c-1984fffa7aa5" containerID="94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c" exitCode=0 Nov 02 23:34:22.067825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.067641 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" Nov 02 23:34:22.067825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.067672 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" event={"ID":"ce8f5119-0c53-41e6-a97c-1984fffa7aa5","Type":"ContainerDied","Data":"94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c"} Nov 02 23:34:22.067825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.067734 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch" event={"ID":"ce8f5119-0c53-41e6-a97c-1984fffa7aa5","Type":"ContainerDied","Data":"123dc306bd80c73d7195a065283b6fcced5c1c8a196be9df98837c542fcc8a92"} Nov 02 23:34:22.067825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.067765 5672 scope.go:122] "RemoveContainer" containerID="94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c" Nov 02 23:34:22.069229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.069210 5672 generic.go:356] "Generic (PLEG): container finished" podID="8d33def5-1207-4710-8abd-04250c8cc5a4" containerID="7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048" exitCode=0 Nov 02 23:34:22.069351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.069254 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" Nov 02 23:34:22.069351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.069330 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" event={"ID":"8d33def5-1207-4710-8abd-04250c8cc5a4","Type":"ContainerDied","Data":"7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048"} Nov 02 23:34:22.069454 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.069388 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw" event={"ID":"8d33def5-1207-4710-8abd-04250c8cc5a4","Type":"ContainerDied","Data":"27a8a50316b422b9208089f0c9cf6ae004e0223fb4a9f8dc85c2cff99f32e61b"} Nov 02 23:34:22.080798 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.080710 5672 scope.go:122] "RemoveContainer" containerID="94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c" Nov 02 23:34:22.081197 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:34:22.081166 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c\": container with ID starting with 94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c not found: ID does not exist" containerID="94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c" Nov 02 23:34:22.081333 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.081204 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c"} err="failed to get container status \"94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c\": rpc error: code = NotFound desc = could not find container \"94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c\": container with ID starting with 94b2a1d621de4aef86449e22890c24bab1174fd558645c6a91fdf7eeadb9e29c not found: ID does not exist" Nov 02 23:34:22.081333 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.081224 5672 scope.go:122] "RemoveContainer" containerID="7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048" Nov 02 23:34:22.090070 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.090052 5672 scope.go:122] "RemoveContainer" containerID="7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048" Nov 02 23:34:22.090357 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:34:22.090330 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048\": container with ID starting with 7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048 not found: ID does not exist" containerID="7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048" Nov 02 23:34:22.090447 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.090360 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048"} err="failed to get container status \"7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048\": rpc error: code = NotFound desc = could not find container \"7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048\": container with ID starting with 7f48d65ce76a0b97066f0ba2d3b0b5288bb4489fa97f1a9e21a8d4f1c6cc4048 not found: ID does not exist" Nov 02 23:34:22.225827 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.225791 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw"] Nov 02 23:34:22.229634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.229609 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-81c3c9-75b4fbc8bc-6psbw"] Nov 02 23:34:22.374118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.374026 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch"] Nov 02 23:34:22.377697 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.377670 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-81c3c9-654965fb8f-s6vch"] Nov 02 23:34:22.712717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.712638 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d33def5-1207-4710-8abd-04250c8cc5a4" path="/var/lib/kubelet/pods/8d33def5-1207-4710-8abd-04250c8cc5a4/volumes" Nov 02 23:34:22.713036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:22.713023 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8f5119-0c53-41e6-a97c-1984fffa7aa5" path="/var/lib/kubelet/pods/ce8f5119-0c53-41e6-a97c-1984fffa7aa5/volumes" Nov 02 23:34:26.020708 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:34:26.020671 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:34:26.495202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:26.495143 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:34:29.325336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.325299 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9"] Nov 02 23:34:29.326002 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.325978 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8d33def5-1207-4710-8abd-04250c8cc5a4" containerName="payload-pre-processing" Nov 02 23:34:29.326081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.326005 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d33def5-1207-4710-8abd-04250c8cc5a4" containerName="payload-pre-processing" Nov 02 23:34:29.326081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.326037 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ce8f5119-0c53-41e6-a97c-1984fffa7aa5" containerName="payload-processing" Nov 02 23:34:29.326081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.326046 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8f5119-0c53-41e6-a97c-1984fffa7aa5" containerName="payload-processing" Nov 02 23:34:29.326081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.326080 5672 cpu_manager.go:414] "containerMap: removing container" podUID="938f8589-bbe7-4ef6-8fe8-43c81f550222" containerName="revoke-keys" Nov 02 23:34:29.326223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.326090 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938f8589-bbe7-4ef6-8fe8-43c81f550222" containerName="revoke-keys" Nov 02 23:34:29.326254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.326246 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8f5119-0c53-41e6-a97c-1984fffa7aa5" containerName="payload-processing" Nov 02 23:34:29.326287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.326265 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="938f8589-bbe7-4ef6-8fe8-43c81f550222" containerName="revoke-keys" Nov 02 23:34:29.326287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.326282 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d33def5-1207-4710-8abd-04250c8cc5a4" containerName="payload-pre-processing" Nov 02 23:34:29.330554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.330530 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:29.381932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.381897 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9"] Nov 02 23:34:29.424452 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.424413 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:34:29.425402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.425374 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:34:29.425402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.425386 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:34:29.425667 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.425528 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:34:29.465481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.465446 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b0fc8e-q6rn9\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:29.465632 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.465507 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b0fc8e-q6rn9\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:29.465700 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.465661 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vqk\" (UniqueName: \"kubernetes.io/projected/90486d35-1f87-46b4-b4ee-23f26ceb47d3-kube-api-access-k6vqk\") pod \"maas-api-revoke-sub-30b0fc8e-q6rn9\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:29.566568 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.566534 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b0fc8e-q6rn9\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:29.566727 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.566585 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b0fc8e-q6rn9\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:29.566727 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.566719 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vqk\" (UniqueName: \"kubernetes.io/projected/90486d35-1f87-46b4-b4ee-23f26ceb47d3-kube-api-access-k6vqk\") pod \"maas-api-revoke-sub-30b0fc8e-q6rn9\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:29.567225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.567203 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b0fc8e-q6rn9\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:29.568993 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.568966 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b0fc8e-q6rn9\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:29.620400 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.620348 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vqk\" (UniqueName: \"kubernetes.io/projected/90486d35-1f87-46b4-b4ee-23f26ceb47d3-kube-api-access-k6vqk\") pod \"maas-api-revoke-sub-30b0fc8e-q6rn9\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:29.642553 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:29.642532 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:30.005059 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:30.005027 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9"] Nov 02 23:34:30.006375 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:34:30.006339 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90486d35_1f87_46b4_b4ee_23f26ceb47d3.slice/crio-b8ba6845c3762a1347ebd0a39122ab630767a8435ee7be7b26aec4001ffa764c WatchSource:0}: Error finding container b8ba6845c3762a1347ebd0a39122ab630767a8435ee7be7b26aec4001ffa764c: Status 404 returned error can't find the container with id b8ba6845c3762a1347ebd0a39122ab630767a8435ee7be7b26aec4001ffa764c Nov 02 23:34:30.125731 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:30.125689 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" event={"ID":"90486d35-1f87-46b4-b4ee-23f26ceb47d3","Type":"ContainerStarted","Data":"b8ba6845c3762a1347ebd0a39122ab630767a8435ee7be7b26aec4001ffa764c"} Nov 02 23:34:31.133920 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:31.133880 5672 generic.go:356] "Generic (PLEG): container finished" podID="90486d35-1f87-46b4-b4ee-23f26ceb47d3" containerID="dfbedee0d8cdd96c677b0414a63cd2ff1f50002308ae8fb458511779b00d7023" exitCode=0 Nov 02 23:34:31.134337 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:31.133921 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" event={"ID":"90486d35-1f87-46b4-b4ee-23f26ceb47d3","Type":"ContainerDied","Data":"dfbedee0d8cdd96c677b0414a63cd2ff1f50002308ae8fb458511779b00d7023"} Nov 02 23:34:32.298473 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:32.298445 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:32.397379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:32.397306 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-xks-ca\") pod \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " Nov 02 23:34:32.397379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:32.397374 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90486d35-1f87-46b4-b4ee-23f26ceb47d3-kube-api-access-k6vqk\" (UniqueName: \"kubernetes.io/projected/90486d35-1f87-46b4-b4ee-23f26ceb47d3-kube-api-access-k6vqk\") pod \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " Nov 02 23:34:32.397562 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:32.397447 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-openshift-ca\") pod \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\" (UID: \"90486d35-1f87-46b4-b4ee-23f26ceb47d3\") " Nov 02 23:34:32.397807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:32.397780 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-openshift-ca" pod "90486d35-1f87-46b4-b4ee-23f26ceb47d3" (UID: "90486d35-1f87-46b4-b4ee-23f26ceb47d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:34:32.399421 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:32.399394 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90486d35-1f87-46b4-b4ee-23f26ceb47d3-kube-api-access-k6vqk" pod "90486d35-1f87-46b4-b4ee-23f26ceb47d3" (UID: "90486d35-1f87-46b4-b4ee-23f26ceb47d3"). InnerVolumeSpecName "kube-api-access-k6vqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:34:32.399510 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:32.399496 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-xks-ca" pod "90486d35-1f87-46b4-b4ee-23f26ceb47d3" (UID: "90486d35-1f87-46b4-b4ee-23f26ceb47d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:34:32.498291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:32.498261 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:32.498291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:32.498285 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6vqk\" (UniqueName: \"kubernetes.io/projected/90486d35-1f87-46b4-b4ee-23f26ceb47d3-kube-api-access-k6vqk\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:32.498291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:32.498295 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90486d35-1f87-46b4-b4ee-23f26ceb47d3-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:33.146181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:33.146068 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" event={"ID":"90486d35-1f87-46b4-b4ee-23f26ceb47d3","Type":"ContainerDied","Data":"b8ba6845c3762a1347ebd0a39122ab630767a8435ee7be7b26aec4001ffa764c"} Nov 02 23:34:33.146181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:33.146129 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8ba6845c3762a1347ebd0a39122ab630767a8435ee7be7b26aec4001ffa764c" Nov 02 23:34:33.146181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:33.146077 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9" Nov 02 23:34:45.148289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.148254 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8"] Nov 02 23:34:45.149318 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.149290 5672 cpu_manager.go:414] "containerMap: removing container" podUID="90486d35-1f87-46b4-b4ee-23f26ceb47d3" containerName="revoke-keys" Nov 02 23:34:45.149318 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.149318 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90486d35-1f87-46b4-b4ee-23f26ceb47d3" containerName="revoke-keys" Nov 02 23:34:45.149765 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.149749 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="90486d35-1f87-46b4-b4ee-23f26ceb47d3" containerName="revoke-keys" Nov 02 23:34:45.153749 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.153723 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:45.249504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.249468 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:34:45.249504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.249505 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:34:45.249782 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.249583 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:34:45.249782 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.249651 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:34:45.252653 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.252624 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8"] Nov 02 23:34:45.343897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.343865 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:45.344081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.343916 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:45.344121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.344071 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5w6r\" (UniqueName: \"kubernetes.io/projected/9dda5427-121c-4ee8-9036-4a7829443a3f-kube-api-access-n5w6r\") pod \"maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:45.445143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.445045 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5w6r\" (UniqueName: \"kubernetes.io/projected/9dda5427-121c-4ee8-9036-4a7829443a3f-kube-api-access-n5w6r\") pod \"maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:45.445564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.445536 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:45.445791 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.445767 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:45.446555 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.446527 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:45.448457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.448437 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:45.498718 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.498679 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5w6r\" (UniqueName: \"kubernetes.io/projected/9dda5427-121c-4ee8-9036-4a7829443a3f-kube-api-access-n5w6r\") pod \"maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:45.769752 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:45.769720 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:46.146664 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:46.146639 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8"] Nov 02 23:34:46.147618 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:34:46.147587 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dda5427_121c_4ee8_9036_4a7829443a3f.slice/crio-320e2e87dcb7a2c36b3f19e996f376da7fb9af8f0d3b8141287f15113014c3e2 WatchSource:0}: Error finding container 320e2e87dcb7a2c36b3f19e996f376da7fb9af8f0d3b8141287f15113014c3e2: Status 404 returned error can't find the container with id 320e2e87dcb7a2c36b3f19e996f376da7fb9af8f0d3b8141287f15113014c3e2 Nov 02 23:34:46.235785 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:46.235756 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" event={"ID":"9dda5427-121c-4ee8-9036-4a7829443a3f","Type":"ContainerStarted","Data":"320e2e87dcb7a2c36b3f19e996f376da7fb9af8f0d3b8141287f15113014c3e2"} Nov 02 23:34:47.243319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:47.243294 5672 generic.go:356] "Generic (PLEG): container finished" podID="9dda5427-121c-4ee8-9036-4a7829443a3f" containerID="586a3642c50e54eafa7bd86da1ace6736883771fcc379937419c60545471cd62" exitCode=0 Nov 02 23:34:47.243662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:47.243379 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" event={"ID":"9dda5427-121c-4ee8-9036-4a7829443a3f","Type":"ContainerDied","Data":"586a3642c50e54eafa7bd86da1ace6736883771fcc379937419c60545471cd62"} Nov 02 23:34:48.414330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.414296 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:48.482190 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.482127 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-openshift-ca\") pod \"9dda5427-121c-4ee8-9036-4a7829443a3f\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " Nov 02 23:34:48.482379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.482295 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dda5427-121c-4ee8-9036-4a7829443a3f-kube-api-access-n5w6r\" (UniqueName: \"kubernetes.io/projected/9dda5427-121c-4ee8-9036-4a7829443a3f-kube-api-access-n5w6r\") pod \"9dda5427-121c-4ee8-9036-4a7829443a3f\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " Nov 02 23:34:48.482379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.482357 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-xks-ca\") pod \"9dda5427-121c-4ee8-9036-4a7829443a3f\" (UID: \"9dda5427-121c-4ee8-9036-4a7829443a3f\") " Nov 02 23:34:48.482548 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.482511 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-openshift-ca" pod "9dda5427-121c-4ee8-9036-4a7829443a3f" (UID: "9dda5427-121c-4ee8-9036-4a7829443a3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:34:48.482784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.482758 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:48.484546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.484520 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dda5427-121c-4ee8-9036-4a7829443a3f-kube-api-access-n5w6r" pod "9dda5427-121c-4ee8-9036-4a7829443a3f" (UID: "9dda5427-121c-4ee8-9036-4a7829443a3f"). InnerVolumeSpecName "kube-api-access-n5w6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:34:48.484745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.484727 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-xks-ca" pod "9dda5427-121c-4ee8-9036-4a7829443a3f" (UID: "9dda5427-121c-4ee8-9036-4a7829443a3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:34:48.583623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.583529 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5w6r\" (UniqueName: \"kubernetes.io/projected/9dda5427-121c-4ee8-9036-4a7829443a3f-kube-api-access-n5w6r\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:48.583623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.583566 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dda5427-121c-4ee8-9036-4a7829443a3f-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:48.894817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.894737 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw"] Nov 02 23:34:48.895296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.895277 5672 cpu_manager.go:414] "containerMap: removing container" podUID="9dda5427-121c-4ee8-9036-4a7829443a3f" containerName="revoke-keys" Nov 02 23:34:48.895296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.895297 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dda5427-121c-4ee8-9036-4a7829443a3f" containerName="revoke-keys" Nov 02 23:34:48.895483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.895467 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dda5427-121c-4ee8-9036-4a7829443a3f" containerName="revoke-keys" Nov 02 23:34:48.898016 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.897998 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:48.953098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.952770 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw"] Nov 02 23:34:48.991198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.986269 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp5dd\" (UniqueName: \"kubernetes.io/projected/ebaef9a0-d3f6-4953-a457-13d84409ffcc-kube-api-access-gp5dd\") pod \"maas-api-revoke-sub-cb1d8584-hgvcw\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:48.991681 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.991655 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb1d8584-hgvcw\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:48.991826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:48.991811 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb1d8584-hgvcw\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:49.092888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:49.092856 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb1d8584-hgvcw\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:49.093087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:49.092894 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb1d8584-hgvcw\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:49.093087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:49.092935 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5dd\" (UniqueName: \"kubernetes.io/projected/ebaef9a0-d3f6-4953-a457-13d84409ffcc-kube-api-access-gp5dd\") pod \"maas-api-revoke-sub-cb1d8584-hgvcw\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:49.093560 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:49.093537 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb1d8584-hgvcw\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:49.095460 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:49.095434 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb1d8584-hgvcw\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:49.146279 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:49.146209 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5dd\" (UniqueName: \"kubernetes.io/projected/ebaef9a0-d3f6-4953-a457-13d84409ffcc-kube-api-access-gp5dd\") pod \"maas-api-revoke-sub-cb1d8584-hgvcw\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:49.222187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:49.222138 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:49.258430 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:49.258401 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" Nov 02 23:34:49.258570 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:49.258394 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8" event={"ID":"9dda5427-121c-4ee8-9036-4a7829443a3f","Type":"ContainerDied","Data":"320e2e87dcb7a2c36b3f19e996f376da7fb9af8f0d3b8141287f15113014c3e2"} Nov 02 23:34:49.258570 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:49.258524 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="320e2e87dcb7a2c36b3f19e996f376da7fb9af8f0d3b8141287f15113014c3e2" Nov 02 23:34:49.590751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:49.590725 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw"] Nov 02 23:34:49.592030 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:34:49.591990 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebaef9a0_d3f6_4953_a457_13d84409ffcc.slice/crio-d1497bc00d68ac622b1448f0a8bee32ea5a18c800456fd6ac34e2fd67c29313b WatchSource:0}: Error finding container d1497bc00d68ac622b1448f0a8bee32ea5a18c800456fd6ac34e2fd67c29313b: Status 404 returned error can't find the container with id d1497bc00d68ac622b1448f0a8bee32ea5a18c800456fd6ac34e2fd67c29313b Nov 02 23:34:50.267317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.267274 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" event={"ID":"ebaef9a0-d3f6-4953-a457-13d84409ffcc","Type":"ContainerStarted","Data":"2de2ce30b44bc8c9e8ddc1313cd638626512262e57e9d9ea848e1a6602733579"} Nov 02 23:34:50.267522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.267339 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" event={"ID":"ebaef9a0-d3f6-4953-a457-13d84409ffcc","Type":"ContainerStarted","Data":"d1497bc00d68ac622b1448f0a8bee32ea5a18c800456fd6ac34e2fd67c29313b"} Nov 02 23:34:50.377326 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.377275 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" podStartSLOduration=2.377260962 podStartE2EDuration="2.377260962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:34:50.374566447 +0000 UTC m=+2483.781878271" watchObservedRunningTime="2026-11-02 23:34:50.377260962 +0000 UTC m=+2483.784572791" Nov 02 23:34:50.655948 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.655909 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk"] Nov 02 23:34:50.659506 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.659481 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:50.712950 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.712910 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8cfq\" (UniqueName: \"kubernetes.io/projected/3fc6a834-a50c-474b-81de-61212576468c-kube-api-access-h8cfq\") pod \"maas-api-revoke-sub-8343b3f2-65hvk\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:50.713124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.712975 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc6a834-a50c-474b-81de-61212576468c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8343b3f2-65hvk\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:50.713235 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.713215 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc6a834-a50c-474b-81de-61212576468c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8343b3f2-65hvk\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:50.715775 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.715747 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk"] Nov 02 23:34:50.814493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.814404 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc6a834-a50c-474b-81de-61212576468c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8343b3f2-65hvk\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:50.814493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.814444 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8cfq\" (UniqueName: \"kubernetes.io/projected/3fc6a834-a50c-474b-81de-61212576468c-kube-api-access-h8cfq\") pod \"maas-api-revoke-sub-8343b3f2-65hvk\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:50.814493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.814470 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc6a834-a50c-474b-81de-61212576468c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8343b3f2-65hvk\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:50.815206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.815183 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc6a834-a50c-474b-81de-61212576468c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8343b3f2-65hvk\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:50.817051 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.817025 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc6a834-a50c-474b-81de-61212576468c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8343b3f2-65hvk\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:50.868327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.868301 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8cfq\" (UniqueName: \"kubernetes.io/projected/3fc6a834-a50c-474b-81de-61212576468c-kube-api-access-h8cfq\") pod \"maas-api-revoke-sub-8343b3f2-65hvk\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:50.974039 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:50.974009 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:51.275051 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:51.275004 5672 generic.go:356] "Generic (PLEG): container finished" podID="ebaef9a0-d3f6-4953-a457-13d84409ffcc" containerID="2de2ce30b44bc8c9e8ddc1313cd638626512262e57e9d9ea848e1a6602733579" exitCode=0 Nov 02 23:34:51.275350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:51.275067 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" event={"ID":"ebaef9a0-d3f6-4953-a457-13d84409ffcc","Type":"ContainerDied","Data":"2de2ce30b44bc8c9e8ddc1313cd638626512262e57e9d9ea848e1a6602733579"} Nov 02 23:34:51.339598 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:51.339577 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk"] Nov 02 23:34:51.341565 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:34:51.341535 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc6a834_a50c_474b_81de_61212576468c.slice/crio-fd609f148706abb95f3076ae5199944b72d907896bb30679b1ac79a5f41072cb WatchSource:0}: Error finding container fd609f148706abb95f3076ae5199944b72d907896bb30679b1ac79a5f41072cb: Status 404 returned error can't find the container with id fd609f148706abb95f3076ae5199944b72d907896bb30679b1ac79a5f41072cb Nov 02 23:34:52.283840 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.283795 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" event={"ID":"3fc6a834-a50c-474b-81de-61212576468c","Type":"ContainerStarted","Data":"d6581234bd8d76a73f2c760fd9ee8fcc6569e79ae0307d2ec15c83cd7708eb45"} Nov 02 23:34:52.284294 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.283852 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" event={"ID":"3fc6a834-a50c-474b-81de-61212576468c","Type":"ContainerStarted","Data":"fd609f148706abb95f3076ae5199944b72d907896bb30679b1ac79a5f41072cb"} Nov 02 23:34:52.392322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.392272 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" podStartSLOduration=2.392251702 podStartE2EDuration="2.392251702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:34:52.388654939 +0000 UTC m=+2485.795966762" watchObservedRunningTime="2026-11-02 23:34:52.392251702 +0000 UTC m=+2485.799563526" Nov 02 23:34:52.416955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.416922 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:52.536704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.536631 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-openshift-ca\") pod \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " Nov 02 23:34:52.536704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.536695 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebaef9a0-d3f6-4953-a457-13d84409ffcc-kube-api-access-gp5dd\" (UniqueName: \"kubernetes.io/projected/ebaef9a0-d3f6-4953-a457-13d84409ffcc-kube-api-access-gp5dd\") pod \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " Nov 02 23:34:52.536932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.536735 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-xks-ca\") pod \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\" (UID: \"ebaef9a0-d3f6-4953-a457-13d84409ffcc\") " Nov 02 23:34:52.537039 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.537009 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-openshift-ca" pod "ebaef9a0-d3f6-4953-a457-13d84409ffcc" (UID: "ebaef9a0-d3f6-4953-a457-13d84409ffcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:34:52.537207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.537182 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:52.538958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.538934 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebaef9a0-d3f6-4953-a457-13d84409ffcc-kube-api-access-gp5dd" pod "ebaef9a0-d3f6-4953-a457-13d84409ffcc" (UID: "ebaef9a0-d3f6-4953-a457-13d84409ffcc"). InnerVolumeSpecName "kube-api-access-gp5dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:34:52.538958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.538947 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-xks-ca" pod "ebaef9a0-d3f6-4953-a457-13d84409ffcc" (UID: "ebaef9a0-d3f6-4953-a457-13d84409ffcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:34:52.638346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.638304 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp5dd\" (UniqueName: \"kubernetes.io/projected/ebaef9a0-d3f6-4953-a457-13d84409ffcc-kube-api-access-gp5dd\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:52.638346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:52.638341 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebaef9a0-d3f6-4953-a457-13d84409ffcc-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:53.292091 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:53.292054 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" Nov 02 23:34:53.292511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:53.292091 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw" event={"ID":"ebaef9a0-d3f6-4953-a457-13d84409ffcc","Type":"ContainerDied","Data":"d1497bc00d68ac622b1448f0a8bee32ea5a18c800456fd6ac34e2fd67c29313b"} Nov 02 23:34:53.292511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:53.292172 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1497bc00d68ac622b1448f0a8bee32ea5a18c800456fd6ac34e2fd67c29313b" Nov 02 23:34:53.294365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:53.293708 5672 generic.go:356] "Generic (PLEG): container finished" podID="3fc6a834-a50c-474b-81de-61212576468c" containerID="d6581234bd8d76a73f2c760fd9ee8fcc6569e79ae0307d2ec15c83cd7708eb45" exitCode=0 Nov 02 23:34:53.294365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:53.293741 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" event={"ID":"3fc6a834-a50c-474b-81de-61212576468c","Type":"ContainerDied","Data":"d6581234bd8d76a73f2c760fd9ee8fcc6569e79ae0307d2ec15c83cd7708eb45"} Nov 02 23:34:54.460588 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:54.460556 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:54.560106 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:54.560014 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc6a834-a50c-474b-81de-61212576468c-kube-api-access-h8cfq\" (UniqueName: \"kubernetes.io/projected/3fc6a834-a50c-474b-81de-61212576468c-kube-api-access-h8cfq\") pod \"3fc6a834-a50c-474b-81de-61212576468c\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " Nov 02 23:34:54.560106 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:54.560098 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc6a834-a50c-474b-81de-61212576468c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc6a834-a50c-474b-81de-61212576468c-maas-api-openshift-ca\") pod \"3fc6a834-a50c-474b-81de-61212576468c\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " Nov 02 23:34:54.560342 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:54.560241 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc6a834-a50c-474b-81de-61212576468c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc6a834-a50c-474b-81de-61212576468c-maas-api-xks-ca\") pod \"3fc6a834-a50c-474b-81de-61212576468c\" (UID: \"3fc6a834-a50c-474b-81de-61212576468c\") " Nov 02 23:34:54.560514 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:54.560472 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc6a834-a50c-474b-81de-61212576468c-maas-api-openshift-ca" pod "3fc6a834-a50c-474b-81de-61212576468c" (UID: "3fc6a834-a50c-474b-81de-61212576468c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:34:54.560661 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:54.560635 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc6a834-a50c-474b-81de-61212576468c-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:54.562362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:54.562335 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc6a834-a50c-474b-81de-61212576468c-kube-api-access-h8cfq" pod "3fc6a834-a50c-474b-81de-61212576468c" (UID: "3fc6a834-a50c-474b-81de-61212576468c"). InnerVolumeSpecName "kube-api-access-h8cfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:34:54.562607 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:54.562588 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc6a834-a50c-474b-81de-61212576468c-maas-api-xks-ca" pod "3fc6a834-a50c-474b-81de-61212576468c" (UID: "3fc6a834-a50c-474b-81de-61212576468c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:34:54.662137 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:54.662080 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc6a834-a50c-474b-81de-61212576468c-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:54.662336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:54.662137 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8cfq\" (UniqueName: \"kubernetes.io/projected/3fc6a834-a50c-474b-81de-61212576468c-kube-api-access-h8cfq\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:34:55.306835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:55.306802 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" Nov 02 23:34:55.307015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:55.306856 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk" event={"ID":"3fc6a834-a50c-474b-81de-61212576468c","Type":"ContainerDied","Data":"fd609f148706abb95f3076ae5199944b72d907896bb30679b1ac79a5f41072cb"} Nov 02 23:34:55.307015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:55.306915 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd609f148706abb95f3076ae5199944b72d907896bb30679b1ac79a5f41072cb" Nov 02 23:34:56.020351 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:34:56.020312 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:34:59.018000 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:34:59.017962 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:35:04.511848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:04.511815 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d"] Nov 02 23:35:04.611478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:04.611448 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l"] Nov 02 23:35:05.079669 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.079635 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh"] Nov 02 23:35:05.080118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.080105 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3fc6a834-a50c-474b-81de-61212576468c" containerName="revoke-keys" Nov 02 23:35:05.080188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.080119 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc6a834-a50c-474b-81de-61212576468c" containerName="revoke-keys" Nov 02 23:35:05.080188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.080129 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ebaef9a0-d3f6-4953-a457-13d84409ffcc" containerName="revoke-keys" Nov 02 23:35:05.080188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.080136 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebaef9a0-d3f6-4953-a457-13d84409ffcc" containerName="revoke-keys" Nov 02 23:35:05.080294 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.080232 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc6a834-a50c-474b-81de-61212576468c" containerName="revoke-keys" Nov 02 23:35:05.080294 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.080240 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebaef9a0-d3f6-4953-a457-13d84409ffcc" containerName="revoke-keys" Nov 02 23:35:05.084853 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.084831 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.168336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.168300 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh"] Nov 02 23:35:05.182397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.182355 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg96x\" (UniqueName: \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-kube-api-access-wg96x\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.182571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.182446 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.182571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.182485 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.182695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.182588 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.182695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.182662 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.182775 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.182726 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.182775 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.182761 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.182851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.182787 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.182912 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.182897 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.224105 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.224075 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz"] Nov 02 23:35:05.240274 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.240251 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:05.281056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.281022 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz"] Nov 02 23:35:05.283745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.283720 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.283857 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.283785 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.283857 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.283811 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.283857 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.283829 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.283857 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.283848 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.284075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.283990 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.284134 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.284078 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg96x\" (UniqueName: \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-kube-api-access-wg96x\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.284134 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.284120 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.284269 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.284181 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.284332 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.284182 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.284402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.284379 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.284463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.284439 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.284527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.284484 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.284702 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.284670 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.286843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.286819 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.287361 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.287340 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.289234 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.289208 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:35:05.289458 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.289440 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:35:05.289579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.289513 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:35:05.289579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.289525 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:35:05.336877 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.336807 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.337339 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.337317 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg96x\" (UniqueName: \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-kube-api-access-wg96x\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.385391 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.385361 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f62f45f2-pspxz\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:05.385391 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.385392 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8psm5\" (UniqueName: \"kubernetes.io/projected/34daf46b-0833-45a6-981b-7b7c545be946-kube-api-access-8psm5\") pod \"maas-api-revoke-sub-f62f45f2-pspxz\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:05.385593 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.385528 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f62f45f2-pspxz\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:05.399111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.399081 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:05.486908 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.486872 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f62f45f2-pspxz\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:05.487041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.486922 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8psm5\" (UniqueName: \"kubernetes.io/projected/34daf46b-0833-45a6-981b-7b7c545be946-kube-api-access-8psm5\") pod \"maas-api-revoke-sub-f62f45f2-pspxz\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:05.487041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.487029 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f62f45f2-pspxz\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:05.487873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.487833 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f62f45f2-pspxz\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:05.489714 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.489693 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f62f45f2-pspxz\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:05.540321 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.540288 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8psm5\" (UniqueName: \"kubernetes.io/projected/34daf46b-0833-45a6-981b-7b7c545be946-kube-api-access-8psm5\") pod \"maas-api-revoke-sub-f62f45f2-pspxz\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:05.554169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.554114 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:05.830557 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.829389 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:05.830557 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.829483 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:05.830557 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.829529 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:05.852776 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.852745 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh"] Nov 02 23:35:05.925453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:05.925296 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz"] Nov 02 23:35:05.925708 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:35:05.925679 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34daf46b_0833_45a6_981b_7b7c545be946.slice/crio-496c820c9bd8507f77874a23bd13eededecee74c94cf8b921cf7180d0330d56c WatchSource:0}: Error finding container 496c820c9bd8507f77874a23bd13eededecee74c94cf8b921cf7180d0330d56c: Status 404 returned error can't find the container with id 496c820c9bd8507f77874a23bd13eededecee74c94cf8b921cf7180d0330d56c Nov 02 23:35:06.383225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:06.383184 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" event={"ID":"afabbfc9-a7bd-43f6-8923-83c13a54d7ac","Type":"ContainerStarted","Data":"bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101"} Nov 02 23:35:06.383385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:06.383269 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" event={"ID":"afabbfc9-a7bd-43f6-8923-83c13a54d7ac","Type":"ContainerStarted","Data":"03c86e22898a7636cf7c87f31937d7a9697262ce7e118ee9baabe50a3e626c4b"} Nov 02 23:35:06.384812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:06.384785 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" event={"ID":"34daf46b-0833-45a6-981b-7b7c545be946","Type":"ContainerStarted","Data":"6d9ef55be6ae6872df7560822c90b2997cc912950f6a58d63309a5eac03121c6"} Nov 02 23:35:06.384951 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:06.384825 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" event={"ID":"34daf46b-0833-45a6-981b-7b7c545be946","Type":"ContainerStarted","Data":"496c820c9bd8507f77874a23bd13eededecee74c94cf8b921cf7180d0330d56c"} Nov 02 23:35:06.399387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:06.399336 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:06.400633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:06.400612 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:35:06.400709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:06.400657 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 02 23:35:06.494433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:06.494381 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" podStartSLOduration=1.494367953 podStartE2EDuration="1.494367953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:06.491708897 +0000 UTC m=+2499.899020729" watchObservedRunningTime="2026-11-02 23:35:06.494367953 +0000 UTC m=+2499.901679783" Nov 02 23:35:06.599384 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:06.599333 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" podStartSLOduration=1.5993191690000002 podStartE2EDuration="1.599319169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:06.597774894 +0000 UTC m=+2500.005086725" watchObservedRunningTime="2026-11-02 23:35:06.599319169 +0000 UTC m=+2500.006631064" Nov 02 23:35:07.392864 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:07.392772 5672 generic.go:356] "Generic (PLEG): container finished" podID="34daf46b-0833-45a6-981b-7b7c545be946" containerID="6d9ef55be6ae6872df7560822c90b2997cc912950f6a58d63309a5eac03121c6" exitCode=0 Nov 02 23:35:07.393015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:07.392860 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" event={"ID":"34daf46b-0833-45a6-981b-7b7c545be946","Type":"ContainerDied","Data":"6d9ef55be6ae6872df7560822c90b2997cc912950f6a58d63309a5eac03121c6"} Nov 02 23:35:07.399585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:07.399556 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:35:07.399716 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:07.399613 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 02 23:35:08.399656 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.399621 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:35:08.400170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.399683 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 02 23:35:08.622312 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.622282 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:08.755223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.724340 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34daf46b-0833-45a6-981b-7b7c545be946-kube-api-access-8psm5\" (UniqueName: \"kubernetes.io/projected/34daf46b-0833-45a6-981b-7b7c545be946-kube-api-access-8psm5\") pod \"34daf46b-0833-45a6-981b-7b7c545be946\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " Nov 02 23:35:08.755223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.724931 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-openshift-ca\") pod \"34daf46b-0833-45a6-981b-7b7c545be946\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " Nov 02 23:35:08.755223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.725309 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-openshift-ca" pod "34daf46b-0833-45a6-981b-7b7c545be946" (UID: "34daf46b-0833-45a6-981b-7b7c545be946"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:08.755223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.725414 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-xks-ca\") pod \"34daf46b-0833-45a6-981b-7b7c545be946\" (UID: \"34daf46b-0833-45a6-981b-7b7c545be946\") " Nov 02 23:35:08.755223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.725822 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:08.769174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.755998 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-xks-ca" pod "34daf46b-0833-45a6-981b-7b7c545be946" (UID: "34daf46b-0833-45a6-981b-7b7c545be946"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:35:08.772423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.772383 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34daf46b-0833-45a6-981b-7b7c545be946-kube-api-access-8psm5" pod "34daf46b-0833-45a6-981b-7b7c545be946" (UID: "34daf46b-0833-45a6-981b-7b7c545be946"). InnerVolumeSpecName "kube-api-access-8psm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:08.859173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.858933 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34daf46b-0833-45a6-981b-7b7c545be946-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:08.859173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:08.858963 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8psm5\" (UniqueName: \"kubernetes.io/projected/34daf46b-0833-45a6-981b-7b7c545be946-kube-api-access-8psm5\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:09.400419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.400377 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:35:09.400836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.400446 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 02 23:35:09.408826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.408799 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" event={"ID":"34daf46b-0833-45a6-981b-7b7c545be946","Type":"ContainerDied","Data":"496c820c9bd8507f77874a23bd13eededecee74c94cf8b921cf7180d0330d56c"} Nov 02 23:35:09.408955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.408833 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz" Nov 02 23:35:09.408955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.408845 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="496c820c9bd8507f77874a23bd13eededecee74c94cf8b921cf7180d0330d56c" Nov 02 23:35:09.512869 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.512833 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" podUID="523f9203-2495-4872-9f45-ab996220c4c7" containerName="payload-processing" containerID="cri-o://c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa" gracePeriod=25 Nov 02 23:35:09.611986 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.611939 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" podUID="baed7942-8752-4757-8a2b-9b3993a91617" containerName="payload-pre-processing" containerID="cri-o://b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a" gracePeriod=25 Nov 02 23:35:09.861560 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.860547 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:35:09.971788 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.971761 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523f9203-2495-4872-9f45-ab996220c4c7-kube-api-access-t2wg9\" (UniqueName: \"kubernetes.io/projected/523f9203-2495-4872-9f45-ab996220c4c7-kube-api-access-t2wg9\") pod \"523f9203-2495-4872-9f45-ab996220c4c7\" (UID: \"523f9203-2495-4872-9f45-ab996220c4c7\") " Nov 02 23:35:09.971927 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.971850 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/523f9203-2495-4872-9f45-ab996220c4c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/523f9203-2495-4872-9f45-ab996220c4c7-plugins-config-volume\") pod \"523f9203-2495-4872-9f45-ab996220c4c7\" (UID: \"523f9203-2495-4872-9f45-ab996220c4c7\") " Nov 02 23:35:09.972222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.972195 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523f9203-2495-4872-9f45-ab996220c4c7-plugins-config-volume" pod "523f9203-2495-4872-9f45-ab996220c4c7" (UID: "523f9203-2495-4872-9f45-ab996220c4c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:09.972380 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.972362 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/523f9203-2495-4872-9f45-ab996220c4c7-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:09.973883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.973856 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523f9203-2495-4872-9f45-ab996220c4c7-kube-api-access-t2wg9" pod "523f9203-2495-4872-9f45-ab996220c4c7" (UID: "523f9203-2495-4872-9f45-ab996220c4c7"). InnerVolumeSpecName "kube-api-access-t2wg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:09.977523 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:09.977504 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:35:10.073559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.073524 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baed7942-8752-4757-8a2b-9b3993a91617-kube-api-access-fvscj\" (UniqueName: \"kubernetes.io/projected/baed7942-8752-4757-8a2b-9b3993a91617-kube-api-access-fvscj\") pod \"baed7942-8752-4757-8a2b-9b3993a91617\" (UID: \"baed7942-8752-4757-8a2b-9b3993a91617\") " Nov 02 23:35:10.073773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.073587 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baed7942-8752-4757-8a2b-9b3993a91617-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baed7942-8752-4757-8a2b-9b3993a91617-plugins-config-volume\") pod \"baed7942-8752-4757-8a2b-9b3993a91617\" (UID: \"baed7942-8752-4757-8a2b-9b3993a91617\") " Nov 02 23:35:10.073881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.073867 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2wg9\" (UniqueName: \"kubernetes.io/projected/523f9203-2495-4872-9f45-ab996220c4c7-kube-api-access-t2wg9\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:10.073976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.073953 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baed7942-8752-4757-8a2b-9b3993a91617-plugins-config-volume" pod "baed7942-8752-4757-8a2b-9b3993a91617" (UID: "baed7942-8752-4757-8a2b-9b3993a91617"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:10.075698 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.075670 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baed7942-8752-4757-8a2b-9b3993a91617-kube-api-access-fvscj" pod "baed7942-8752-4757-8a2b-9b3993a91617" (UID: "baed7942-8752-4757-8a2b-9b3993a91617"). InnerVolumeSpecName "kube-api-access-fvscj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:10.174948 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.174905 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvscj\" (UniqueName: \"kubernetes.io/projected/baed7942-8752-4757-8a2b-9b3993a91617-kube-api-access-fvscj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:10.174948 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.174941 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baed7942-8752-4757-8a2b-9b3993a91617-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:10.400322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.400290 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:35:10.400491 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.400340 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 02 23:35:10.417478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.417446 5672 generic.go:356] "Generic (PLEG): container finished" podID="baed7942-8752-4757-8a2b-9b3993a91617" containerID="b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a" exitCode=0 Nov 02 23:35:10.417641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.417490 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" Nov 02 23:35:10.417641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.417527 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" event={"ID":"baed7942-8752-4757-8a2b-9b3993a91617","Type":"ContainerDied","Data":"b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a"} Nov 02 23:35:10.417641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.417585 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l" event={"ID":"baed7942-8752-4757-8a2b-9b3993a91617","Type":"ContainerDied","Data":"d9966b1a73c592312e68ac390ff50c61e307ebe8961516f62cb02a83c502d6f0"} Nov 02 23:35:10.417641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.417617 5672 scope.go:122] "RemoveContainer" containerID="b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a" Nov 02 23:35:10.419256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.419219 5672 generic.go:356] "Generic (PLEG): container finished" podID="523f9203-2495-4872-9f45-ab996220c4c7" containerID="c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa" exitCode=0 Nov 02 23:35:10.419370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.419254 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" Nov 02 23:35:10.419370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.419256 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" event={"ID":"523f9203-2495-4872-9f45-ab996220c4c7","Type":"ContainerDied","Data":"c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa"} Nov 02 23:35:10.419468 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.419389 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d" event={"ID":"523f9203-2495-4872-9f45-ab996220c4c7","Type":"ContainerDied","Data":"7deab8819e60a867225c198392c94960b0909b701ba7268aa4f1e9cf496f76c0"} Nov 02 23:35:10.457969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.457944 5672 scope.go:122] "RemoveContainer" containerID="b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a" Nov 02 23:35:10.458350 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:35:10.458321 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a\": container with ID starting with b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a not found: ID does not exist" containerID="b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a" Nov 02 23:35:10.458411 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.458360 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a"} err="failed to get container status \"b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a\": rpc error: code = NotFound desc = could not find container \"b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a\": container with ID starting with b7bbf3fd5b6d797341e2e689c37fb32b3ebe7636e91c1a374aab76801eecb36a not found: ID does not exist" Nov 02 23:35:10.458411 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.458379 5672 scope.go:122] "RemoveContainer" containerID="c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa" Nov 02 23:35:10.468027 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.468001 5672 scope.go:122] "RemoveContainer" containerID="c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa" Nov 02 23:35:10.468361 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:35:10.468340 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa\": container with ID starting with c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa not found: ID does not exist" containerID="c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa" Nov 02 23:35:10.468447 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.468369 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa"} err="failed to get container status \"c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa\": rpc error: code = NotFound desc = could not find container \"c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa\": container with ID starting with c9d5119ba25340b5bd6882ec150072c5387a419dc9dcb1d0fb8b86ad6ee55bfa not found: ID does not exist" Nov 02 23:35:10.572376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.572337 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l"] Nov 02 23:35:10.576397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.576366 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fb5fa2-5c6b586d4c-mkl9l"] Nov 02 23:35:10.677843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.677745 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d"] Nov 02 23:35:10.680958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.680934 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fb5fa2-ddffcd4d7-znn5d"] Nov 02 23:35:10.712443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.712414 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523f9203-2495-4872-9f45-ab996220c4c7" path="/var/lib/kubelet/pods/523f9203-2495-4872-9f45-ab996220c4c7/volumes" Nov 02 23:35:10.712875 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:10.712860 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baed7942-8752-4757-8a2b-9b3993a91617" path="/var/lib/kubelet/pods/baed7942-8752-4757-8a2b-9b3993a91617/volumes" Nov 02 23:35:11.399572 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:11.399533 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:35:11.399773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:11.399588 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 02 23:35:12.400251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:12.400219 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 02 23:35:12.400641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:12.400270 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 02 23:35:13.567723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:13.567691 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:13.568249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:13.568042 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:13.568926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:13.568908 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:35:22.730594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.730560 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8"] Nov 02 23:35:22.731164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.731021 5672 cpu_manager.go:414] "containerMap: removing container" podUID="523f9203-2495-4872-9f45-ab996220c4c7" containerName="payload-processing" Nov 02 23:35:22.731164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.731032 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523f9203-2495-4872-9f45-ab996220c4c7" containerName="payload-processing" Nov 02 23:35:22.731164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.731051 5672 cpu_manager.go:414] "containerMap: removing container" podUID="baed7942-8752-4757-8a2b-9b3993a91617" containerName="payload-pre-processing" Nov 02 23:35:22.731164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.731056 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baed7942-8752-4757-8a2b-9b3993a91617" containerName="payload-pre-processing" Nov 02 23:35:22.731164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.731066 5672 cpu_manager.go:414] "containerMap: removing container" podUID="34daf46b-0833-45a6-981b-7b7c545be946" containerName="revoke-keys" Nov 02 23:35:22.731164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.731071 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34daf46b-0833-45a6-981b-7b7c545be946" containerName="revoke-keys" Nov 02 23:35:22.731164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.731161 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="baed7942-8752-4757-8a2b-9b3993a91617" containerName="payload-pre-processing" Nov 02 23:35:22.731457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.731172 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="34daf46b-0833-45a6-981b-7b7c545be946" containerName="revoke-keys" Nov 02 23:35:22.731457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.731192 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="523f9203-2495-4872-9f45-ab996220c4c7" containerName="payload-processing" Nov 02 23:35:22.734206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.734188 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:22.787441 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.787410 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:35:22.819574 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.819531 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6dada84-cdvz8\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:22.819724 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.819628 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6dada84-cdvz8\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:22.819784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.819722 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz5zt\" (UniqueName: \"kubernetes.io/projected/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-kube-api-access-bz5zt\") pod \"maas-api-revoke-sub-b6dada84-cdvz8\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:22.834139 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.834111 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:35:22.834139 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.834133 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:35:22.834139 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.834119 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:35:22.838896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.838867 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8"] Nov 02 23:35:22.921109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.921077 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6dada84-cdvz8\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:22.921260 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.921143 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz5zt\" (UniqueName: \"kubernetes.io/projected/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-kube-api-access-bz5zt\") pod \"maas-api-revoke-sub-b6dada84-cdvz8\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:22.921302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.921265 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6dada84-cdvz8\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:22.921725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.921707 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6dada84-cdvz8\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:22.923794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.923767 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6dada84-cdvz8\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:22.974685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:22.974651 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz5zt\" (UniqueName: \"kubernetes.io/projected/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-kube-api-access-bz5zt\") pod \"maas-api-revoke-sub-b6dada84-cdvz8\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:23.051602 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:23.051521 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:23.420099 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:23.420069 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8"] Nov 02 23:35:23.421383 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:35:23.421344 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbfd6f9_581a_4136_86a8_f5ec01c7454b.slice/crio-5edb1b66e7e118f0d6601ccda6a74fd01ec29b1ab0611bfbc0fd888df860b29a WatchSource:0}: Error finding container 5edb1b66e7e118f0d6601ccda6a74fd01ec29b1ab0611bfbc0fd888df860b29a: Status 404 returned error can't find the container with id 5edb1b66e7e118f0d6601ccda6a74fd01ec29b1ab0611bfbc0fd888df860b29a Nov 02 23:35:23.542990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:23.542955 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" event={"ID":"9cbfd6f9-581a-4136-86a8-f5ec01c7454b","Type":"ContainerStarted","Data":"5edb1b66e7e118f0d6601ccda6a74fd01ec29b1ab0611bfbc0fd888df860b29a"} Nov 02 23:35:24.552778 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:24.552740 5672 generic.go:356] "Generic (PLEG): container finished" podID="9cbfd6f9-581a-4136-86a8-f5ec01c7454b" containerID="72657f7d516f9e0de2ebff901eefbba6feda675f8160e6e611ac70ffdff188dc" exitCode=0 Nov 02 23:35:24.553223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:24.552818 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" event={"ID":"9cbfd6f9-581a-4136-86a8-f5ec01c7454b","Type":"ContainerDied","Data":"72657f7d516f9e0de2ebff901eefbba6feda675f8160e6e611ac70ffdff188dc"} Nov 02 23:35:25.704091 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:25.704062 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:25.752289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:25.752259 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-kube-api-access-bz5zt\" (UniqueName: \"kubernetes.io/projected/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-kube-api-access-bz5zt\") pod \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " Nov 02 23:35:25.752446 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:25.752349 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-xks-ca\") pod \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " Nov 02 23:35:25.752515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:25.752493 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-openshift-ca\") pod \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\" (UID: \"9cbfd6f9-581a-4136-86a8-f5ec01c7454b\") " Nov 02 23:35:25.752969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:25.752933 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-openshift-ca" pod "9cbfd6f9-581a-4136-86a8-f5ec01c7454b" (UID: "9cbfd6f9-581a-4136-86a8-f5ec01c7454b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:25.754895 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:25.754871 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-kube-api-access-bz5zt" pod "9cbfd6f9-581a-4136-86a8-f5ec01c7454b" (UID: "9cbfd6f9-581a-4136-86a8-f5ec01c7454b"). InnerVolumeSpecName "kube-api-access-bz5zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:25.755015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:25.754989 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-xks-ca" pod "9cbfd6f9-581a-4136-86a8-f5ec01c7454b" (UID: "9cbfd6f9-581a-4136-86a8-f5ec01c7454b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:35:25.854587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:25.854494 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz5zt\" (UniqueName: \"kubernetes.io/projected/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-kube-api-access-bz5zt\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:25.854587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:25.854523 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:25.854587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:25.854534 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cbfd6f9-581a-4136-86a8-f5ec01c7454b-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:26.020058 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:35:26.020012 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:35:26.571038 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:26.571000 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" Nov 02 23:35:26.571038 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:26.571017 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8" event={"ID":"9cbfd6f9-581a-4136-86a8-f5ec01c7454b","Type":"ContainerDied","Data":"5edb1b66e7e118f0d6601ccda6a74fd01ec29b1ab0611bfbc0fd888df860b29a"} Nov 02 23:35:26.571297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:26.571068 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5edb1b66e7e118f0d6601ccda6a74fd01ec29b1ab0611bfbc0fd888df860b29a" Nov 02 23:35:38.943017 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:38.942936 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz"] Nov 02 23:35:38.943453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:38.943310 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" containerID="cri-o://43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab" gracePeriod=30 Nov 02 23:35:39.029529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.029498 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl"] Nov 02 23:35:39.029981 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.029968 5672 cpu_manager.go:414] "containerMap: removing container" podUID="9cbfd6f9-581a-4136-86a8-f5ec01c7454b" containerName="revoke-keys" Nov 02 23:35:39.030026 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.029983 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbfd6f9-581a-4136-86a8-f5ec01c7454b" containerName="revoke-keys" Nov 02 23:35:39.030116 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.030104 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbfd6f9-581a-4136-86a8-f5ec01c7454b" containerName="revoke-keys" Nov 02 23:35:39.035043 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.035019 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:39.087333 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.087299 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl"] Nov 02 23:35:39.087884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.087859 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:35:39.088014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.087939 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:35:39.088085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.088012 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:35:39.088085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.088057 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:35:39.116362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.116318 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-981724-k89xl\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:39.116538 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.116409 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-981724-k89xl\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:39.116602 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.116538 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fszp\" (UniqueName: \"kubernetes.io/projected/e3e08cdb-d262-43e2-82d0-17e03fe193bd-kube-api-access-7fszp\") pod \"maas-api-revoke-keys-e2e-ipp-a-981724-k89xl\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:39.217384 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.217298 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-981724-k89xl\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:39.217384 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.217344 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-981724-k89xl\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:39.217624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.217388 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fszp\" (UniqueName: \"kubernetes.io/projected/e3e08cdb-d262-43e2-82d0-17e03fe193bd-kube-api-access-7fszp\") pod \"maas-api-revoke-keys-e2e-ipp-a-981724-k89xl\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:39.218036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.218015 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-981724-k89xl\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:39.219732 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.219710 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-981724-k89xl\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:39.271001 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.270970 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fszp\" (UniqueName: \"kubernetes.io/projected/e3e08cdb-d262-43e2-82d0-17e03fe193bd-kube-api-access-7fszp\") pod \"maas-api-revoke-keys-e2e-ipp-a-981724-k89xl\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:39.389173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.389121 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:39.761802 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:39.761768 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl"] Nov 02 23:35:39.762742 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:35:39.762711 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e08cdb_d262_43e2_82d0_17e03fe193bd.slice/crio-c227acb7e41a2eb21fc38fafab299b47d7e2bdc7508dfea8712b8ccfe01542ce WatchSource:0}: Error finding container c227acb7e41a2eb21fc38fafab299b47d7e2bdc7508dfea8712b8ccfe01542ce: Status 404 returned error can't find the container with id c227acb7e41a2eb21fc38fafab299b47d7e2bdc7508dfea8712b8ccfe01542ce Nov 02 23:35:40.671670 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:40.671632 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" event={"ID":"e3e08cdb-d262-43e2-82d0-17e03fe193bd","Type":"ContainerStarted","Data":"0b9478f4c978f87829581889ad81e5427cba10f201d57dd894e05590181a5cee"} Nov 02 23:35:40.672169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:40.671688 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" event={"ID":"e3e08cdb-d262-43e2-82d0-17e03fe193bd","Type":"ContainerStarted","Data":"c227acb7e41a2eb21fc38fafab299b47d7e2bdc7508dfea8712b8ccfe01542ce"} Nov 02 23:35:40.780542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:40.780489 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" podStartSLOduration=2.780473696 podStartE2EDuration="2.780473696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:40.777832347 +0000 UTC m=+2534.185144177" watchObservedRunningTime="2026-11-02 23:35:40.780473696 +0000 UTC m=+2534.187785526" Nov 02 23:35:41.684327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:41.684284 5672 generic.go:356] "Generic (PLEG): container finished" podID="e3e08cdb-d262-43e2-82d0-17e03fe193bd" containerID="0b9478f4c978f87829581889ad81e5427cba10f201d57dd894e05590181a5cee" exitCode=0 Nov 02 23:35:41.684841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:41.684512 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" event={"ID":"e3e08cdb-d262-43e2-82d0-17e03fe193bd","Type":"ContainerDied","Data":"0b9478f4c978f87829581889ad81e5427cba10f201d57dd894e05590181a5cee"} Nov 02 23:35:42.638239 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:42.638202 5672 patch_prober.go:29] interesting pod/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:35:42.638461 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:42.638267 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:35:42.854383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:42.854353 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:42.958664 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:42.958584 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-openshift-ca\") pod \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " Nov 02 23:35:42.958839 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:42.958740 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e08cdb-d262-43e2-82d0-17e03fe193bd-kube-api-access-7fszp\" (UniqueName: \"kubernetes.io/projected/e3e08cdb-d262-43e2-82d0-17e03fe193bd-kube-api-access-7fszp\") pod \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " Nov 02 23:35:42.958839 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:42.958802 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-xks-ca\") pod \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\" (UID: \"e3e08cdb-d262-43e2-82d0-17e03fe193bd\") " Nov 02 23:35:42.959033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:42.959005 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-openshift-ca" pod "e3e08cdb-d262-43e2-82d0-17e03fe193bd" (UID: "e3e08cdb-d262-43e2-82d0-17e03fe193bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:42.959241 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:42.959220 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:42.961568 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:42.961527 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e08cdb-d262-43e2-82d0-17e03fe193bd-kube-api-access-7fszp" pod "e3e08cdb-d262-43e2-82d0-17e03fe193bd" (UID: "e3e08cdb-d262-43e2-82d0-17e03fe193bd"). InnerVolumeSpecName "kube-api-access-7fszp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:42.962085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:42.962054 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-xks-ca" pod "e3e08cdb-d262-43e2-82d0-17e03fe193bd" (UID: "e3e08cdb-d262-43e2-82d0-17e03fe193bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:35:43.061019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:43.060979 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fszp\" (UniqueName: \"kubernetes.io/projected/e3e08cdb-d262-43e2-82d0-17e03fe193bd-kube-api-access-7fszp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:43.061019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:43.061018 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e08cdb-d262-43e2-82d0-17e03fe193bd-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:43.698627 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:43.698597 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" Nov 02 23:35:43.698812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:43.698631 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl" event={"ID":"e3e08cdb-d262-43e2-82d0-17e03fe193bd","Type":"ContainerDied","Data":"c227acb7e41a2eb21fc38fafab299b47d7e2bdc7508dfea8712b8ccfe01542ce"} Nov 02 23:35:43.698812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:43.698679 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c227acb7e41a2eb21fc38fafab299b47d7e2bdc7508dfea8712b8ccfe01542ce" Nov 02 23:35:44.271811 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.270793 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:35:44.394858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.393606 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istiod-ca-cert\") pod \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " Nov 02 23:35:44.394858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.393664 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-socket\") pod \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " Nov 02 23:35:44.394858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.393742 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-kube-api-access-2k7zx\" (UniqueName: \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-kube-api-access-2k7zx\") pod \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " Nov 02 23:35:44.394858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.393787 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-credential-socket\") pod \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " Nov 02 23:35:44.394858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.393852 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-envoy\") pod \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " Nov 02 23:35:44.394858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.393890 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-certs\") pod \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " Nov 02 23:35:44.394858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.393943 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-data\") pod \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " Nov 02 23:35:44.394858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.394002 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-podinfo\") pod \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " Nov 02 23:35:44.394858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.394058 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-token\" (UniqueName: \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-token\") pod \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\" (UID: \"589ff327-0656-4cc4-ae2f-0c5c2c4d0059\") " Nov 02 23:35:44.395893 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.395856 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-credential-socket" pod "589ff327-0656-4cc4-ae2f-0c5c2c4d0059" (UID: "589ff327-0656-4cc4-ae2f-0c5c2c4d0059"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:44.398590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.397651 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-data" pod "589ff327-0656-4cc4-ae2f-0c5c2c4d0059" (UID: "589ff327-0656-4cc4-ae2f-0c5c2c4d0059"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:44.398590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.397955 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-certs" pod "589ff327-0656-4cc4-ae2f-0c5c2c4d0059" (UID: "589ff327-0656-4cc4-ae2f-0c5c2c4d0059"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:44.399054 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.399015 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istiod-ca-cert" pod "589ff327-0656-4cc4-ae2f-0c5c2c4d0059" (UID: "589ff327-0656-4cc4-ae2f-0c5c2c4d0059"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:35:44.399381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.399362 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-socket" pod "589ff327-0656-4cc4-ae2f-0c5c2c4d0059" (UID: "589ff327-0656-4cc4-ae2f-0c5c2c4d0059"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:44.407331 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.407300 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-token" pod "589ff327-0656-4cc4-ae2f-0c5c2c4d0059" (UID: "589ff327-0656-4cc4-ae2f-0c5c2c4d0059"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:44.422559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.422522 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-kube-api-access-2k7zx" pod "589ff327-0656-4cc4-ae2f-0c5c2c4d0059" (UID: "589ff327-0656-4cc4-ae2f-0c5c2c4d0059"). InnerVolumeSpecName "kube-api-access-2k7zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:35:44.425742 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.425296 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-envoy" pod "589ff327-0656-4cc4-ae2f-0c5c2c4d0059" (UID: "589ff327-0656-4cc4-ae2f-0c5c2c4d0059"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:35:44.434345 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.431587 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-podinfo" pod "589ff327-0656-4cc4-ae2f-0c5c2c4d0059" (UID: "589ff327-0656-4cc4-ae2f-0c5c2c4d0059"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:35:44.446687 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.446649 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:35:44.495197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.495085 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k7zx\" (UniqueName: \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-kube-api-access-2k7zx\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:44.495197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.495113 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:44.495197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.495123 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:44.495197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.495132 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:44.495197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.495141 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:44.495197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.495179 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:44.495197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.495194 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:44.495197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.495206 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:44.495639 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.495216 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/589ff327-0656-4cc4-ae2f-0c5c2c4d0059-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:35:44.707092 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.707061 5672 generic.go:356] "Generic (PLEG): container finished" podID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerID="43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab" exitCode=0 Nov 02 23:35:44.707303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.707207 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" Nov 02 23:35:44.711978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.711951 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" event={"ID":"589ff327-0656-4cc4-ae2f-0c5c2c4d0059","Type":"ContainerDied","Data":"43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab"} Nov 02 23:35:44.712118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.711989 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz" event={"ID":"589ff327-0656-4cc4-ae2f-0c5c2c4d0059","Type":"ContainerDied","Data":"83ba1cfe8750db6e488595c6af79c7ad441b550620bfd0957c442eac4884410b"} Nov 02 23:35:44.712118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.712013 5672 scope.go:122] "RemoveContainer" containerID="43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab" Nov 02 23:35:44.723094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.723072 5672 scope.go:122] "RemoveContainer" containerID="8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6" Nov 02 23:35:44.732520 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.732496 5672 scope.go:122] "RemoveContainer" containerID="43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab" Nov 02 23:35:44.732812 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:35:44.732792 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab\": container with ID starting with 43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab not found: ID does not exist" containerID="43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab" Nov 02 23:35:44.732870 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.732821 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab"} err="failed to get container status \"43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab\": rpc error: code = NotFound desc = could not find container \"43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab\": container with ID starting with 43e01933378fc032b35f4991e4f61dcc31abd691d75b0c8cda82313001df2cab not found: ID does not exist" Nov 02 23:35:44.732870 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.732841 5672 scope.go:122] "RemoveContainer" containerID="8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6" Nov 02 23:35:44.733102 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:35:44.733083 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6\": container with ID starting with 8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6 not found: ID does not exist" containerID="8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6" Nov 02 23:35:44.733182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.733108 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6"} err="failed to get container status \"8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6\": rpc error: code = NotFound desc = could not find container \"8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6\": container with ID starting with 8d39620da4df91857996e6894f56b9dc95c3caec4e5da059c57f1e3093ceb2a6 not found: ID does not exist" Nov 02 23:35:44.861915 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.861836 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz"] Nov 02 23:35:44.908464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:44.908421 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a3909-openshift-default-7d9c4dbff7-mdtgz"] Nov 02 23:35:46.712502 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:46.712471 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" path="/var/lib/kubelet/pods/589ff327-0656-4cc4-ae2f-0c5c2c4d0059/volumes" Nov 02 23:35:50.425312 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.425277 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr"] Nov 02 23:35:50.425782 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.425709 5672 cpu_manager.go:414] "containerMap: removing container" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" Nov 02 23:35:50.425782 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.425721 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" Nov 02 23:35:50.425782 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.425745 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e3e08cdb-d262-43e2-82d0-17e03fe193bd" containerName="revoke-keys" Nov 02 23:35:50.425782 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.425751 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e08cdb-d262-43e2-82d0-17e03fe193bd" containerName="revoke-keys" Nov 02 23:35:50.426014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.425825 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" Nov 02 23:35:50.426014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.425834 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e08cdb-d262-43e2-82d0-17e03fe193bd" containerName="revoke-keys" Nov 02 23:35:50.431173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.431122 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.560239 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.560196 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96526185-0fff-403d-83df-146ecdbd24e0-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.560426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.560257 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96526185-0fff-403d-83df-146ecdbd24e0-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.560426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.560287 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ccbm\" (UniqueName: \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-kube-api-access-5ccbm\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.560426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.560330 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.560597 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.560433 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.560597 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.560469 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.560597 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.560527 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.560754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.560603 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.560754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.560635 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.616348 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.616320 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr"] Nov 02 23:35:50.662210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.662168 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.662442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.662245 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.662442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.662273 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.662442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.662344 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96526185-0fff-403d-83df-146ecdbd24e0-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.662442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.662391 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96526185-0fff-403d-83df-146ecdbd24e0-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.662442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.662418 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ccbm\" (UniqueName: \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-kube-api-access-5ccbm\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.662719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.662459 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.662719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.662558 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.662719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.662592 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.662719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.662657 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.662719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.662701 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.663083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.663059 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96526185-0fff-403d-83df-146ecdbd24e0-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.663192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.663132 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.663259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.663219 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.664852 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.664824 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.665110 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.665090 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96526185-0fff-403d-83df-146ecdbd24e0-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.715776 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.715715 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.715776 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.715720 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ccbm\" (UniqueName: \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-kube-api-access-5ccbm\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:50.744998 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:50.744977 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:51.164892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:51.164867 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr"] Nov 02 23:35:51.166748 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:35:51.166712 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96526185_0fff_403d_83df_146ecdbd24e0.slice/crio-68f270e966e148dca99f170243996acad3657be50be1a618acb1e34e0f7e42ac WatchSource:0}: Error finding container 68f270e966e148dca99f170243996acad3657be50be1a618acb1e34e0f7e42ac: Status 404 returned error can't find the container with id 68f270e966e148dca99f170243996acad3657be50be1a618acb1e34e0f7e42ac Nov 02 23:35:51.169390 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:51.169359 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:51.169510 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:51.169437 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:51.169510 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:51.169479 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:35:51.766543 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:51.766500 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" event={"ID":"96526185-0fff-403d-83df-146ecdbd24e0","Type":"ContainerStarted","Data":"6021d421cf0aa3fbb993abf198674d231cc549a8bac2f6effb822f95ca397ae5"} Nov 02 23:35:51.767166 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:51.767124 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" event={"ID":"96526185-0fff-403d-83df-146ecdbd24e0","Type":"ContainerStarted","Data":"68f270e966e148dca99f170243996acad3657be50be1a618acb1e34e0f7e42ac"} Nov 02 23:35:51.928067 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:51.928000 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" podStartSLOduration=1.927980787 podStartE2EDuration="1.927980787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:51.92380828 +0000 UTC m=+2545.331120109" watchObservedRunningTime="2026-11-02 23:35:51.927980787 +0000 UTC m=+2545.335292620" Nov 02 23:35:52.745468 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:52.745432 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:52.750487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:52.750454 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 23:35:52.750732 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:52.750706 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" podUID="96526185-0fff-403d-83df-146ecdbd24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 23:35:53.749181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:53.748532 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 23:35:53.749181 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:53.748593 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" podUID="96526185-0fff-403d-83df-146ecdbd24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 23:35:53.863085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:53.863047 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8"] Nov 02 23:35:53.898971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:53.898940 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81c3c9-9kgs8"] Nov 02 23:35:54.713325 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:54.713292 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dda5427-121c-4ee8-9036-4a7829443a3f" path="/var/lib/kubelet/pods/9dda5427-121c-4ee8-9036-4a7829443a3f/volumes" Nov 02 23:35:54.745538 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:54.745503 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 23:35:54.745791 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:54.745761 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" podUID="96526185-0fff-403d-83df-146ecdbd24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 23:35:55.745559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:55.745520 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 23:35:55.746204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:55.746169 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" podUID="96526185-0fff-403d-83df-146ecdbd24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 23:35:56.021244 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:35:56.021159 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:35:56.745539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:56.745500 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 23:35:56.746174 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:56.745816 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" podUID="96526185-0fff-403d-83df-146ecdbd24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 23:35:57.746140 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:57.746107 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 23:35:57.746319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:57.746180 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" podUID="96526185-0fff-403d-83df-146ecdbd24e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 23:35:58.909542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:58.909508 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:58.910291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:58.910264 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:35:58.911100 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:35:58.911079 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 02 23:36:06.076604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:06.076572 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz"] Nov 02 23:36:06.184186 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:06.184130 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b"] Nov 02 23:36:11.077303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.077260 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" podUID="5f3c779c-88f5-4a05-b467-aaa0307d0f56" containerName="payload-processing" containerID="cri-o://bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f" gracePeriod=25 Nov 02 23:36:11.185291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.185246 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" podUID="8b8fac42-a418-4dfe-aad2-1e54e723a8e0" containerName="payload-pre-processing" containerID="cri-o://ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc" gracePeriod=25 Nov 02 23:36:11.190279 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:36:11.190259 5672 logging.go:55] [core] [Channel #1992 SubChannel #1993]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.145:50052", ServerName: "10.134.0.145:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.145:50052: connect: connection refused" Nov 02 23:36:11.190497 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:36:11.190259 5672 logging.go:55] [core] [Channel #1991 SubChannel #1994]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.144:50052", ServerName: "10.134.0.144:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.144:50052: connect: connection refused" Nov 02 23:36:11.346076 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.346010 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:36:11.415798 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.415765 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3c779c-88f5-4a05-b467-aaa0307d0f56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3c779c-88f5-4a05-b467-aaa0307d0f56-plugins-config-volume\") pod \"5f3c779c-88f5-4a05-b467-aaa0307d0f56\" (UID: \"5f3c779c-88f5-4a05-b467-aaa0307d0f56\") " Nov 02 23:36:11.416014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.415879 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3c779c-88f5-4a05-b467-aaa0307d0f56-kube-api-access-mp9hx\" (UniqueName: \"kubernetes.io/projected/5f3c779c-88f5-4a05-b467-aaa0307d0f56-kube-api-access-mp9hx\") pod \"5f3c779c-88f5-4a05-b467-aaa0307d0f56\" (UID: \"5f3c779c-88f5-4a05-b467-aaa0307d0f56\") " Nov 02 23:36:11.416417 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.416346 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3c779c-88f5-4a05-b467-aaa0307d0f56-plugins-config-volume" pod "5f3c779c-88f5-4a05-b467-aaa0307d0f56" (UID: "5f3c779c-88f5-4a05-b467-aaa0307d0f56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:11.419264 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.419236 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3c779c-88f5-4a05-b467-aaa0307d0f56-kube-api-access-mp9hx" pod "5f3c779c-88f5-4a05-b467-aaa0307d0f56" (UID: "5f3c779c-88f5-4a05-b467-aaa0307d0f56"). InnerVolumeSpecName "kube-api-access-mp9hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:11.451694 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.451673 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:36:11.517770 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.517734 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-kube-api-access-wjmtv\" (UniqueName: \"kubernetes.io/projected/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-kube-api-access-wjmtv\") pod \"8b8fac42-a418-4dfe-aad2-1e54e723a8e0\" (UID: \"8b8fac42-a418-4dfe-aad2-1e54e723a8e0\") " Nov 02 23:36:11.517914 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.517799 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-plugins-config-volume\") pod \"8b8fac42-a418-4dfe-aad2-1e54e723a8e0\" (UID: \"8b8fac42-a418-4dfe-aad2-1e54e723a8e0\") " Nov 02 23:36:11.518096 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.518082 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3c779c-88f5-4a05-b467-aaa0307d0f56-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:11.518218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.518097 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp9hx\" (UniqueName: \"kubernetes.io/projected/5f3c779c-88f5-4a05-b467-aaa0307d0f56-kube-api-access-mp9hx\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:11.518218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.518191 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-plugins-config-volume" pod "8b8fac42-a418-4dfe-aad2-1e54e723a8e0" (UID: "8b8fac42-a418-4dfe-aad2-1e54e723a8e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:11.519844 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.519825 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-kube-api-access-wjmtv" pod "8b8fac42-a418-4dfe-aad2-1e54e723a8e0" (UID: "8b8fac42-a418-4dfe-aad2-1e54e723a8e0"). InnerVolumeSpecName "kube-api-access-wjmtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:11.618968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.618889 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjmtv\" (UniqueName: \"kubernetes.io/projected/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-kube-api-access-wjmtv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:11.618968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.618917 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8fac42-a418-4dfe-aad2-1e54e723a8e0-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:11.900564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.900478 5672 generic.go:356] "Generic (PLEG): container finished" podID="5f3c779c-88f5-4a05-b467-aaa0307d0f56" containerID="bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f" exitCode=0 Nov 02 23:36:11.900564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.900525 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" Nov 02 23:36:11.900812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.900569 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" event={"ID":"5f3c779c-88f5-4a05-b467-aaa0307d0f56","Type":"ContainerDied","Data":"bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f"} Nov 02 23:36:11.900812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.900629 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" event={"ID":"5f3c779c-88f5-4a05-b467-aaa0307d0f56","Type":"ContainerDied","Data":"987de021ecc1b9e9686f9626d3a92273308ba4cc33ab4dcd605932d370f7a118"} Nov 02 23:36:11.900812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.900659 5672 scope.go:122] "RemoveContainer" containerID="bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f" Nov 02 23:36:11.902439 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.902415 5672 generic.go:356] "Generic (PLEG): container finished" podID="8b8fac42-a418-4dfe-aad2-1e54e723a8e0" containerID="ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc" exitCode=0 Nov 02 23:36:11.902578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.902444 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" event={"ID":"8b8fac42-a418-4dfe-aad2-1e54e723a8e0","Type":"ContainerDied","Data":"ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc"} Nov 02 23:36:11.902578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.902455 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" Nov 02 23:36:11.902578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.902477 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" event={"ID":"8b8fac42-a418-4dfe-aad2-1e54e723a8e0","Type":"ContainerDied","Data":"238746e72b78d2e08cbcd95fdd82743e31aeec6b14ea2c11e4afa647c0cbd625"} Nov 02 23:36:11.910925 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.910909 5672 scope.go:122] "RemoveContainer" containerID="bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f" Nov 02 23:36:11.911231 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:36:11.911193 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f\": container with ID starting with bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f not found: ID does not exist" containerID="bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f" Nov 02 23:36:11.911231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.911217 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f"} err="failed to get container status \"bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f\": rpc error: code = NotFound desc = could not find container \"bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f\": container with ID starting with bde873005ae4021900c943b00e28477964df54c87124230a9062e636455c6a6f not found: ID does not exist" Nov 02 23:36:11.911334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.911233 5672 scope.go:122] "RemoveContainer" containerID="ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc" Nov 02 23:36:11.919591 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.919570 5672 scope.go:122] "RemoveContainer" containerID="ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc" Nov 02 23:36:11.919859 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:36:11.919838 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc\": container with ID starting with ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc not found: ID does not exist" containerID="ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc" Nov 02 23:36:11.919914 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:11.919864 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc"} err="failed to get container status \"ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc\": rpc error: code = NotFound desc = could not find container \"ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc\": container with ID starting with ec00dbc75b4f0dad96da4ae9f060a1d470632a9aa34c5742252cf14e7cfa12bc not found: ID does not exist" Nov 02 23:36:12.056690 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:12.056659 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b"] Nov 02 23:36:12.061290 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:12.061262 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b"] Nov 02 23:36:12.160039 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:12.159963 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz"] Nov 02 23:36:12.164268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:12.164239 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz"] Nov 02 23:36:12.190681 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:12.190649 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-1c14e6-696b698b65-zxzgz" podUID="5f3c779c-88f5-4a05-b467-aaa0307d0f56" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.145:50052\" within 1s: context deadline exceeded" Nov 02 23:36:12.190796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:12.190751 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1c14e6-649f69489f-vzm5b" podUID="8b8fac42-a418-4dfe-aad2-1e54e723a8e0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.144:50052\" within 1s: context deadline exceeded" Nov 02 23:36:12.190796 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:36:12.190787 5672 logging.go:55] [core] [Channel #1991 SubChannel #1994]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.144:50052", ServerName: "10.134.0.144:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.144:50052: operation was canceled" Nov 02 23:36:12.389860 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:12.389827 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:36:12.712718 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:12.712676 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3c779c-88f5-4a05-b467-aaa0307d0f56" path="/var/lib/kubelet/pods/5f3c779c-88f5-4a05-b467-aaa0307d0f56/volumes" Nov 02 23:36:12.713163 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:12.713132 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8fac42-a418-4dfe-aad2-1e54e723a8e0" path="/var/lib/kubelet/pods/8b8fac42-a418-4dfe-aad2-1e54e723a8e0/volumes" Nov 02 23:36:16.732434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.732400 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg"] Nov 02 23:36:16.732917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.732803 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8b8fac42-a418-4dfe-aad2-1e54e723a8e0" containerName="payload-pre-processing" Nov 02 23:36:16.732917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.732814 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8fac42-a418-4dfe-aad2-1e54e723a8e0" containerName="payload-pre-processing" Nov 02 23:36:16.732917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.732837 5672 cpu_manager.go:414] "containerMap: removing container" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" Nov 02 23:36:16.732917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.732842 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" Nov 02 23:36:16.732917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.732865 5672 cpu_manager.go:414] "containerMap: removing container" podUID="5f3c779c-88f5-4a05-b467-aaa0307d0f56" containerName="payload-processing" Nov 02 23:36:16.732917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.732870 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3c779c-88f5-4a05-b467-aaa0307d0f56" containerName="payload-processing" Nov 02 23:36:16.733268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.732929 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3c779c-88f5-4a05-b467-aaa0307d0f56" containerName="payload-processing" Nov 02 23:36:16.733268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.732940 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8fac42-a418-4dfe-aad2-1e54e723a8e0" containerName="payload-pre-processing" Nov 02 23:36:16.733268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.732952 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="589ff327-0656-4cc4-ae2f-0c5c2c4d0059" containerName="istio-proxy" Nov 02 23:36:16.737837 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.737815 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:16.789102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.789075 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg"] Nov 02 23:36:16.789246 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.789082 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8fd099-2bpmg\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:16.789246 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.789232 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8fd099-2bpmg\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:16.789327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.789286 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvn7n\" (UniqueName: \"kubernetes.io/projected/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-kube-api-access-gvn7n\") pod \"maas-api-revoke-sub-0a8fd099-2bpmg\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:16.832112 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.832081 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:36:16.832279 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.832164 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:36:16.832782 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.832766 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:36:16.832819 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.832783 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:36:16.890477 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.890451 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn7n\" (UniqueName: \"kubernetes.io/projected/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-kube-api-access-gvn7n\") pod \"maas-api-revoke-sub-0a8fd099-2bpmg\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:16.890644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.890522 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8fd099-2bpmg\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:16.890644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.890581 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8fd099-2bpmg\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:16.891329 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.891306 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8fd099-2bpmg\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:16.893015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.892991 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8fd099-2bpmg\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:16.944083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:16.944057 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvn7n\" (UniqueName: \"kubernetes.io/projected/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-kube-api-access-gvn7n\") pod \"maas-api-revoke-sub-0a8fd099-2bpmg\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:17.051090 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:17.051024 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:17.418607 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:17.418581 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg"] Nov 02 23:36:17.947618 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:17.947570 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" event={"ID":"c84d7488-bd24-4afa-bf9a-a79776fbb3b4","Type":"ContainerStarted","Data":"f6ddb9e66b43e39b9f566bcd27603f88990bbf744f9186b62a21b5a1c881d26d"} Nov 02 23:36:17.947972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:17.947637 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" event={"ID":"c84d7488-bd24-4afa-bf9a-a79776fbb3b4","Type":"ContainerStarted","Data":"02c9e06532d85a82c84577f9bf8ee45233c52d834dc712d9d037cda503f5658e"} Nov 02 23:36:18.058084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:18.057989 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" podStartSLOduration=2.057976277 podStartE2EDuration="2.057976277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:36:18.056763588 +0000 UTC m=+2571.464075417" watchObservedRunningTime="2026-11-02 23:36:18.057976277 +0000 UTC m=+2571.465288143" Nov 02 23:36:18.954574 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:18.954535 5672 generic.go:356] "Generic (PLEG): container finished" podID="c84d7488-bd24-4afa-bf9a-a79776fbb3b4" containerID="f6ddb9e66b43e39b9f566bcd27603f88990bbf744f9186b62a21b5a1c881d26d" exitCode=0 Nov 02 23:36:18.954981 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:18.954622 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" event={"ID":"c84d7488-bd24-4afa-bf9a-a79776fbb3b4","Type":"ContainerDied","Data":"f6ddb9e66b43e39b9f566bcd27603f88990bbf744f9186b62a21b5a1c881d26d"} Nov 02 23:36:20.124586 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.124559 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:20.228177 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.228138 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-openshift-ca\") pod \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " Nov 02 23:36:20.228377 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.228355 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-xks-ca\") pod \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " Nov 02 23:36:20.228459 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.228445 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-kube-api-access-gvn7n\" (UniqueName: \"kubernetes.io/projected/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-kube-api-access-gvn7n\") pod \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\" (UID: \"c84d7488-bd24-4afa-bf9a-a79776fbb3b4\") " Nov 02 23:36:20.228513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.228463 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-openshift-ca" pod "c84d7488-bd24-4afa-bf9a-a79776fbb3b4" (UID: "c84d7488-bd24-4afa-bf9a-a79776fbb3b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:20.228832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.228817 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:20.231020 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.230994 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-xks-ca" pod "c84d7488-bd24-4afa-bf9a-a79776fbb3b4" (UID: "c84d7488-bd24-4afa-bf9a-a79776fbb3b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:36:20.231696 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.231670 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-kube-api-access-gvn7n" pod "c84d7488-bd24-4afa-bf9a-a79776fbb3b4" (UID: "c84d7488-bd24-4afa-bf9a-a79776fbb3b4"). InnerVolumeSpecName "kube-api-access-gvn7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:20.330470 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.330435 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvn7n\" (UniqueName: \"kubernetes.io/projected/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-kube-api-access-gvn7n\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:20.330470 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.330472 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84d7488-bd24-4afa-bf9a-a79776fbb3b4-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:20.968326 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.968292 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" Nov 02 23:36:20.968483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.968327 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg" event={"ID":"c84d7488-bd24-4afa-bf9a-a79776fbb3b4","Type":"ContainerDied","Data":"02c9e06532d85a82c84577f9bf8ee45233c52d834dc712d9d037cda503f5658e"} Nov 02 23:36:20.968483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:20.968374 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02c9e06532d85a82c84577f9bf8ee45233c52d834dc712d9d037cda503f5658e" Nov 02 23:36:26.027224 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:36:26.027184 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:36:32.220364 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.220330 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq"] Nov 02 23:36:32.220987 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.220966 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c84d7488-bd24-4afa-bf9a-a79776fbb3b4" containerName="revoke-keys" Nov 02 23:36:32.221080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.220990 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84d7488-bd24-4afa-bf9a-a79776fbb3b4" containerName="revoke-keys" Nov 02 23:36:32.221138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.221106 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84d7488-bd24-4afa-bf9a-a79776fbb3b4" containerName="revoke-keys" Nov 02 23:36:32.225777 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.225754 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:32.278470 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.278437 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq"] Nov 02 23:36:32.320610 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.320582 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:36:32.320610 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.320605 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:36:32.320806 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.320584 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:36:32.321170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.321133 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:36:32.358287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.358263 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d3a6c40-v7gwq\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:32.358438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.358405 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx7pl\" (UniqueName: \"kubernetes.io/projected/c5c12035-8647-4b3a-bb55-f5338cbe0fee-kube-api-access-bx7pl\") pod \"maas-api-revoke-sub-6d3a6c40-v7gwq\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:32.358481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.358455 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d3a6c40-v7gwq\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:32.459522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.459493 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx7pl\" (UniqueName: \"kubernetes.io/projected/c5c12035-8647-4b3a-bb55-f5338cbe0fee-kube-api-access-bx7pl\") pod \"maas-api-revoke-sub-6d3a6c40-v7gwq\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:32.459666 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.459541 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d3a6c40-v7gwq\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:32.459666 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.459643 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d3a6c40-v7gwq\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:32.460378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.460356 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d3a6c40-v7gwq\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:32.461819 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.461791 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d3a6c40-v7gwq\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:32.512750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.512692 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx7pl\" (UniqueName: \"kubernetes.io/projected/c5c12035-8647-4b3a-bb55-f5338cbe0fee-kube-api-access-bx7pl\") pod \"maas-api-revoke-sub-6d3a6c40-v7gwq\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:32.540955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.540929 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:32.915822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:32.909927 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq"] Nov 02 23:36:33.044032 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:33.044003 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" event={"ID":"c5c12035-8647-4b3a-bb55-f5338cbe0fee","Type":"ContainerStarted","Data":"1ae4f2fecc6be88b03254d26efae172aff6fddd2f26dbc4c229578d7b08e652b"} Nov 02 23:36:34.051929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:34.051833 5672 generic.go:356] "Generic (PLEG): container finished" podID="c5c12035-8647-4b3a-bb55-f5338cbe0fee" containerID="4a2ec73bb2bfc77570d455524d9bea3d4ce7742ae1efb95fae252e82410a58f3" exitCode=0 Nov 02 23:36:34.052325 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:34.051917 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" event={"ID":"c5c12035-8647-4b3a-bb55-f5338cbe0fee","Type":"ContainerDied","Data":"4a2ec73bb2bfc77570d455524d9bea3d4ce7742ae1efb95fae252e82410a58f3"} Nov 02 23:36:35.208924 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:35.208899 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:35.288214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:35.288184 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c12035-8647-4b3a-bb55-f5338cbe0fee-kube-api-access-bx7pl\" (UniqueName: \"kubernetes.io/projected/c5c12035-8647-4b3a-bb55-f5338cbe0fee-kube-api-access-bx7pl\") pod \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " Nov 02 23:36:35.288411 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:35.288333 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-openshift-ca\") pod \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " Nov 02 23:36:35.288411 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:35.288405 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-xks-ca\") pod \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\" (UID: \"c5c12035-8647-4b3a-bb55-f5338cbe0fee\") " Nov 02 23:36:35.288743 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:35.288717 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-openshift-ca" pod "c5c12035-8647-4b3a-bb55-f5338cbe0fee" (UID: "c5c12035-8647-4b3a-bb55-f5338cbe0fee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:35.288843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:35.288820 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:35.290511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:35.290477 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-xks-ca" pod "c5c12035-8647-4b3a-bb55-f5338cbe0fee" (UID: "c5c12035-8647-4b3a-bb55-f5338cbe0fee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:36:35.290629 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:35.290484 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c12035-8647-4b3a-bb55-f5338cbe0fee-kube-api-access-bx7pl" pod "c5c12035-8647-4b3a-bb55-f5338cbe0fee" (UID: "c5c12035-8647-4b3a-bb55-f5338cbe0fee"). InnerVolumeSpecName "kube-api-access-bx7pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:35.390230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:35.390159 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c12035-8647-4b3a-bb55-f5338cbe0fee-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:35.390230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:35.390186 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx7pl\" (UniqueName: \"kubernetes.io/projected/c5c12035-8647-4b3a-bb55-f5338cbe0fee-kube-api-access-bx7pl\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:36.065913 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:36.065875 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" event={"ID":"c5c12035-8647-4b3a-bb55-f5338cbe0fee","Type":"ContainerDied","Data":"1ae4f2fecc6be88b03254d26efae172aff6fddd2f26dbc4c229578d7b08e652b"} Nov 02 23:36:36.065913 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:36.065914 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq" Nov 02 23:36:36.066123 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:36.065931 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ae4f2fecc6be88b03254d26efae172aff6fddd2f26dbc4c229578d7b08e652b" Nov 02 23:36:38.403815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:38.403785 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl"] Nov 02 23:36:38.406556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:38.406528 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-981724-k89xl"] Nov 02 23:36:38.712371 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:38.712291 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e08cdb-d262-43e2-82d0-17e03fe193bd" path="/var/lib/kubelet/pods/e3e08cdb-d262-43e2-82d0-17e03fe193bd/volumes" Nov 02 23:36:43.619094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:43.619062 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:36:47.772020 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.771984 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2"] Nov 02 23:36:47.772680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.772659 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c5c12035-8647-4b3a-bb55-f5338cbe0fee" containerName="revoke-keys" Nov 02 23:36:47.772680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.772683 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c12035-8647-4b3a-bb55-f5338cbe0fee" containerName="revoke-keys" Nov 02 23:36:47.772964 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.772845 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c12035-8647-4b3a-bb55-f5338cbe0fee" containerName="revoke-keys" Nov 02 23:36:47.777173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.777136 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:47.827495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.827468 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2"] Nov 02 23:36:47.842042 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.842016 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brs4\" (UniqueName: \"kubernetes.io/projected/39f86f02-c5ac-45a7-b39f-2a13973ac864-kube-api-access-9brs4\") pod \"maas-api-revoke-sub-1807fa51-zb7m2\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:47.842192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.842167 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1807fa51-zb7m2\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:47.842257 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.842212 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1807fa51-zb7m2\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:47.869813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.869746 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:36:47.869813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.869753 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:36:47.869813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.869781 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:36:47.871755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.871738 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:36:47.943310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.943284 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brs4\" (UniqueName: \"kubernetes.io/projected/39f86f02-c5ac-45a7-b39f-2a13973ac864-kube-api-access-9brs4\") pod \"maas-api-revoke-sub-1807fa51-zb7m2\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:47.943449 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.943354 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1807fa51-zb7m2\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:47.943449 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.943382 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1807fa51-zb7m2\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:47.944187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.944133 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1807fa51-zb7m2\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:47.946051 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:47.946002 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1807fa51-zb7m2\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:48.002961 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:48.001278 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brs4\" (UniqueName: \"kubernetes.io/projected/39f86f02-c5ac-45a7-b39f-2a13973ac864-kube-api-access-9brs4\") pod \"maas-api-revoke-sub-1807fa51-zb7m2\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:48.013853 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:48.013828 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp"] Nov 02 23:36:48.014053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:48.014027 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" containerID="cri-o://daed650db1f729e39703a0558839d434d0e7fedce00dae977eabe5e52be73d2b" gracePeriod=30 Nov 02 23:36:48.089990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:48.089906 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:48.510651 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:48.510619 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2"] Nov 02 23:36:48.512240 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:36:48.512213 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f86f02_c5ac_45a7_b39f_2a13973ac864.slice/crio-f50ed17345a788fc6cd7d317db6e0f4bb2f243bc9ee2672b62f53e9c325861af WatchSource:0}: Error finding container f50ed17345a788fc6cd7d317db6e0f4bb2f243bc9ee2672b62f53e9c325861af: Status 404 returned error can't find the container with id f50ed17345a788fc6cd7d317db6e0f4bb2f243bc9ee2672b62f53e9c325861af Nov 02 23:36:49.181176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:49.181063 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" event={"ID":"39f86f02-c5ac-45a7-b39f-2a13973ac864","Type":"ContainerStarted","Data":"fa52e786ca94aec720c0f6d8ad4cf4db6585158f09afd8813aa578a53a136a73"} Nov 02 23:36:49.181176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:49.181121 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" event={"ID":"39f86f02-c5ac-45a7-b39f-2a13973ac864","Type":"ContainerStarted","Data":"f50ed17345a788fc6cd7d317db6e0f4bb2f243bc9ee2672b62f53e9c325861af"} Nov 02 23:36:49.333262 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:49.333209 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" podStartSLOduration=2.333194326 podStartE2EDuration="2.333194326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:36:49.289229806 +0000 UTC m=+2602.696541635" watchObservedRunningTime="2026-11-02 23:36:49.333194326 +0000 UTC m=+2602.740506154" Nov 02 23:36:49.412214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:49.410469 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q"] Nov 02 23:36:49.412214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:49.411568 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" containerID="cri-o://ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c" gracePeriod=30 Nov 02 23:36:50.189879 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:50.189841 5672 generic.go:356] "Generic (PLEG): container finished" podID="39f86f02-c5ac-45a7-b39f-2a13973ac864" containerID="fa52e786ca94aec720c0f6d8ad4cf4db6585158f09afd8813aa578a53a136a73" exitCode=0 Nov 02 23:36:50.190428 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:50.190037 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" event={"ID":"39f86f02-c5ac-45a7-b39f-2a13973ac864","Type":"ContainerDied","Data":"fa52e786ca94aec720c0f6d8ad4cf4db6585158f09afd8813aa578a53a136a73"} Nov 02 23:36:51.359949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:51.359891 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:51.389054 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:51.389023 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-openshift-ca\") pod \"39f86f02-c5ac-45a7-b39f-2a13973ac864\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " Nov 02 23:36:51.389310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:51.389290 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f86f02-c5ac-45a7-b39f-2a13973ac864-kube-api-access-9brs4\" (UniqueName: \"kubernetes.io/projected/39f86f02-c5ac-45a7-b39f-2a13973ac864-kube-api-access-9brs4\") pod \"39f86f02-c5ac-45a7-b39f-2a13973ac864\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " Nov 02 23:36:51.389423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:51.389406 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-xks-ca\") pod \"39f86f02-c5ac-45a7-b39f-2a13973ac864\" (UID: \"39f86f02-c5ac-45a7-b39f-2a13973ac864\") " Nov 02 23:36:51.389504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:51.389475 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-openshift-ca" pod "39f86f02-c5ac-45a7-b39f-2a13973ac864" (UID: "39f86f02-c5ac-45a7-b39f-2a13973ac864"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:51.389829 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:51.389812 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:51.391466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:51.391440 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f86f02-c5ac-45a7-b39f-2a13973ac864-kube-api-access-9brs4" pod "39f86f02-c5ac-45a7-b39f-2a13973ac864" (UID: "39f86f02-c5ac-45a7-b39f-2a13973ac864"). InnerVolumeSpecName "kube-api-access-9brs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:51.391669 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:51.391652 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-xks-ca" pod "39f86f02-c5ac-45a7-b39f-2a13973ac864" (UID: "39f86f02-c5ac-45a7-b39f-2a13973ac864"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:36:51.490919 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:51.490883 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f86f02-c5ac-45a7-b39f-2a13973ac864-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:51.490919 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:51.490915 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brs4\" (UniqueName: \"kubernetes.io/projected/39f86f02-c5ac-45a7-b39f-2a13973ac864-kube-api-access-9brs4\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:52.205385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:52.205357 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" Nov 02 23:36:52.205708 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:52.205356 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2" event={"ID":"39f86f02-c5ac-45a7-b39f-2a13973ac864","Type":"ContainerDied","Data":"f50ed17345a788fc6cd7d317db6e0f4bb2f243bc9ee2672b62f53e9c325861af"} Nov 02 23:36:52.205837 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:52.205736 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f50ed17345a788fc6cd7d317db6e0f4bb2f243bc9ee2672b62f53e9c325861af" Nov 02 23:36:53.214308 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.214273 5672 generic.go:356] "Generic (PLEG): container finished" podID="70176952-4b2e-4939-98cd-62d60951971d" containerID="daed650db1f729e39703a0558839d434d0e7fedce00dae977eabe5e52be73d2b" exitCode=0 Nov 02 23:36:53.214711 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.214321 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" event={"ID":"70176952-4b2e-4939-98cd-62d60951971d","Type":"ContainerDied","Data":"daed650db1f729e39703a0558839d434d0e7fedce00dae977eabe5e52be73d2b"} Nov 02 23:36:53.214711 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.214420 5672 scope.go:122] "RemoveContainer" containerID="2a8502de80f8e38977d13a28b58bab2db9deb5ae5dec88ea619d48171ddfd2c9" Nov 02 23:36:53.308310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.308281 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:36:53.414137 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414067 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70176952-4b2e-4939-98cd-62d60951971d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70176952-4b2e-4939-98cd-62d60951971d-istiod-ca-cert\") pod \"70176952-4b2e-4939-98cd-62d60951971d\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " Nov 02 23:36:53.414137 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414130 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-socket\") pod \"70176952-4b2e-4939-98cd-62d60951971d\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " Nov 02 23:36:53.414405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414232 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-envoy\") pod \"70176952-4b2e-4939-98cd-62d60951971d\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " Nov 02 23:36:53.414405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414257 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-istio-token\" (UniqueName: \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-istio-token\") pod \"70176952-4b2e-4939-98cd-62d60951971d\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " Nov 02 23:36:53.414405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414285 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-kube-api-access-6gdmg\" (UniqueName: \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-kube-api-access-6gdmg\") pod \"70176952-4b2e-4939-98cd-62d60951971d\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " Nov 02 23:36:53.414405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414310 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70176952-4b2e-4939-98cd-62d60951971d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70176952-4b2e-4939-98cd-62d60951971d-istio-podinfo\") pod \"70176952-4b2e-4939-98cd-62d60951971d\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " Nov 02 23:36:53.414405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414343 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-credential-socket\") pod \"70176952-4b2e-4939-98cd-62d60951971d\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " Nov 02 23:36:53.414673 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414496 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-socket" pod "70176952-4b2e-4939-98cd-62d60951971d" (UID: "70176952-4b2e-4939-98cd-62d60951971d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:53.414673 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414504 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-certs\") pod \"70176952-4b2e-4939-98cd-62d60951971d\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " Nov 02 23:36:53.414673 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414554 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70176952-4b2e-4939-98cd-62d60951971d-istiod-ca-cert" pod "70176952-4b2e-4939-98cd-62d60951971d" (UID: "70176952-4b2e-4939-98cd-62d60951971d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:53.414673 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414575 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-data\") pod \"70176952-4b2e-4939-98cd-62d60951971d\" (UID: \"70176952-4b2e-4939-98cd-62d60951971d\") " Nov 02 23:36:53.414873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414722 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-certs" pod "70176952-4b2e-4939-98cd-62d60951971d" (UID: "70176952-4b2e-4939-98cd-62d60951971d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:53.414873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.414830 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-credential-socket" pod "70176952-4b2e-4939-98cd-62d60951971d" (UID: "70176952-4b2e-4939-98cd-62d60951971d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:53.415040 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.415021 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-data" pod "70176952-4b2e-4939-98cd-62d60951971d" (UID: "70176952-4b2e-4939-98cd-62d60951971d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:53.415279 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.415252 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:53.415370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.415282 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:53.415370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.415296 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70176952-4b2e-4939-98cd-62d60951971d-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:53.415370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.415311 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:53.415370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.415325 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:53.416755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.416722 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70176952-4b2e-4939-98cd-62d60951971d-istio-podinfo" pod "70176952-4b2e-4939-98cd-62d60951971d" (UID: "70176952-4b2e-4939-98cd-62d60951971d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:36:53.416876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.416754 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-istio-token" pod "70176952-4b2e-4939-98cd-62d60951971d" (UID: "70176952-4b2e-4939-98cd-62d60951971d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:53.416931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.416915 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-envoy" pod "70176952-4b2e-4939-98cd-62d60951971d" (UID: "70176952-4b2e-4939-98cd-62d60951971d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:53.416970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.416929 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-kube-api-access-6gdmg" pod "70176952-4b2e-4939-98cd-62d60951971d" (UID: "70176952-4b2e-4939-98cd-62d60951971d"). InnerVolumeSpecName "kube-api-access-6gdmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:53.516650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.516620 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70176952-4b2e-4939-98cd-62d60951971d-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:53.516650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.516645 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:53.516650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.516655 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gdmg\" (UniqueName: \"kubernetes.io/projected/70176952-4b2e-4939-98cd-62d60951971d-kube-api-access-6gdmg\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:53.516876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:53.516663 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70176952-4b2e-4939-98cd-62d60951971d-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:54.029129 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.029086 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj"] Nov 02 23:36:54.029437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.029388 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" containerID="cri-o://2754069ce4878620abcf94b60cbd83d89c43207d85be35d7024f4eb2261671b6" gracePeriod=30 Nov 02 23:36:54.221838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.221805 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" event={"ID":"70176952-4b2e-4939-98cd-62d60951971d","Type":"ContainerDied","Data":"6f1ac12ac3fb37cb67e459c8492dcac60df935f0bfcb4c4402b86a5ebaaa9cd4"} Nov 02 23:36:54.221838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.221825 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp" Nov 02 23:36:54.222428 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.221864 5672 scope.go:122] "RemoveContainer" containerID="daed650db1f729e39703a0558839d434d0e7fedce00dae977eabe5e52be73d2b" Nov 02 23:36:54.376225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.376130 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp"] Nov 02 23:36:54.381819 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.381793 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-db5d87-openshift-default-76bc785845-z6xbp"] Nov 02 23:36:54.686382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.686334 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:36:54.712820 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.712793 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70176952-4b2e-4939-98cd-62d60951971d" path="/var/lib/kubelet/pods/70176952-4b2e-4939-98cd-62d60951971d/volumes" Nov 02 23:36:54.738770 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.738744 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-credential-socket\") pod \"0083ae5e-d949-47ce-8716-fd952f06f490\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " Nov 02 23:36:54.738933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.738846 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-istio-token\" (UniqueName: \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-istio-token\") pod \"0083ae5e-d949-47ce-8716-fd952f06f490\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " Nov 02 23:36:54.738933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.738871 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-kube-api-access-sgbcp\" (UniqueName: \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-kube-api-access-sgbcp\") pod \"0083ae5e-d949-47ce-8716-fd952f06f490\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " Nov 02 23:36:54.738933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.738907 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0083ae5e-d949-47ce-8716-fd952f06f490-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0083ae5e-d949-47ce-8716-fd952f06f490-istio-podinfo\") pod \"0083ae5e-d949-47ce-8716-fd952f06f490\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " Nov 02 23:36:54.739107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.738944 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-data\") pod \"0083ae5e-d949-47ce-8716-fd952f06f490\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " Nov 02 23:36:54.739107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.738977 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-envoy\") pod \"0083ae5e-d949-47ce-8716-fd952f06f490\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " Nov 02 23:36:54.739107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.739006 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-credential-socket" pod "0083ae5e-d949-47ce-8716-fd952f06f490" (UID: "0083ae5e-d949-47ce-8716-fd952f06f490"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:54.739107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.739066 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0083ae5e-d949-47ce-8716-fd952f06f490-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0083ae5e-d949-47ce-8716-fd952f06f490-istiod-ca-cert\") pod \"0083ae5e-d949-47ce-8716-fd952f06f490\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " Nov 02 23:36:54.739337 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.739141 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-socket\") pod \"0083ae5e-d949-47ce-8716-fd952f06f490\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " Nov 02 23:36:54.739337 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.739204 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-certs\") pod \"0083ae5e-d949-47ce-8716-fd952f06f490\" (UID: \"0083ae5e-d949-47ce-8716-fd952f06f490\") " Nov 02 23:36:54.739337 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.739246 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-data" pod "0083ae5e-d949-47ce-8716-fd952f06f490" (UID: "0083ae5e-d949-47ce-8716-fd952f06f490"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:54.739827 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.739654 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:54.739827 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.739684 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:54.739827 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.739743 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-socket" pod "0083ae5e-d949-47ce-8716-fd952f06f490" (UID: "0083ae5e-d949-47ce-8716-fd952f06f490"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:54.739827 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.739781 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-certs" pod "0083ae5e-d949-47ce-8716-fd952f06f490" (UID: "0083ae5e-d949-47ce-8716-fd952f06f490"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:54.740118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.739936 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0083ae5e-d949-47ce-8716-fd952f06f490-istiod-ca-cert" pod "0083ae5e-d949-47ce-8716-fd952f06f490" (UID: "0083ae5e-d949-47ce-8716-fd952f06f490"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:54.741744 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.741723 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-envoy" pod "0083ae5e-d949-47ce-8716-fd952f06f490" (UID: "0083ae5e-d949-47ce-8716-fd952f06f490"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:54.741962 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.741942 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-kube-api-access-sgbcp" pod "0083ae5e-d949-47ce-8716-fd952f06f490" (UID: "0083ae5e-d949-47ce-8716-fd952f06f490"). InnerVolumeSpecName "kube-api-access-sgbcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:54.742198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.742178 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-istio-token" pod "0083ae5e-d949-47ce-8716-fd952f06f490" (UID: "0083ae5e-d949-47ce-8716-fd952f06f490"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:54.742198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.742191 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0083ae5e-d949-47ce-8716-fd952f06f490-istio-podinfo" pod "0083ae5e-d949-47ce-8716-fd952f06f490" (UID: "0083ae5e-d949-47ce-8716-fd952f06f490"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:36:54.840515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.840481 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:54.840515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.840507 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbcp\" (UniqueName: \"kubernetes.io/projected/0083ae5e-d949-47ce-8716-fd952f06f490-kube-api-access-sgbcp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:54.840515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.840517 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0083ae5e-d949-47ce-8716-fd952f06f490-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:54.840725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.840525 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:54.840725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.840533 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0083ae5e-d949-47ce-8716-fd952f06f490-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:54.840725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.840542 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:54.840725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:54.840550 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0083ae5e-d949-47ce-8716-fd952f06f490-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:55.229448 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.229416 5672 generic.go:356] "Generic (PLEG): container finished" podID="0083ae5e-d949-47ce-8716-fd952f06f490" containerID="ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c" exitCode=0 Nov 02 23:36:55.229935 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.229477 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" Nov 02 23:36:55.229935 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.229487 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" event={"ID":"0083ae5e-d949-47ce-8716-fd952f06f490","Type":"ContainerDied","Data":"ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c"} Nov 02 23:36:55.229935 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.229540 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q" event={"ID":"0083ae5e-d949-47ce-8716-fd952f06f490","Type":"ContainerDied","Data":"e17780ca2f798832b21ba2519f4dd87197be3aa170cefcd53eda3943c5ba4135"} Nov 02 23:36:55.229935 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.229562 5672 scope.go:122] "RemoveContainer" containerID="ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c" Nov 02 23:36:55.241362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.241343 5672 scope.go:122] "RemoveContainer" containerID="478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a" Nov 02 23:36:55.250582 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.250562 5672 scope.go:122] "RemoveContainer" containerID="ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c" Nov 02 23:36:55.250823 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:36:55.250803 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c\": container with ID starting with ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c not found: ID does not exist" containerID="ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c" Nov 02 23:36:55.250869 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.250833 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c"} err="failed to get container status \"ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c\": rpc error: code = NotFound desc = could not find container \"ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c\": container with ID starting with ad92958e9646e99fdaa92672c577147a429879d79db390f9802206c3f50fbe3c not found: ID does not exist" Nov 02 23:36:55.250869 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.250850 5672 scope.go:122] "RemoveContainer" containerID="478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a" Nov 02 23:36:55.251073 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:36:55.251059 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a\": container with ID starting with 478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a not found: ID does not exist" containerID="478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a" Nov 02 23:36:55.251115 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.251078 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a"} err="failed to get container status \"478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a\": rpc error: code = NotFound desc = could not find container \"478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a\": container with ID starting with 478c0b546d1a1040e690314ec5aa55c0b094e2f4b1a445ebbe0f415374f93b9a not found: ID does not exist" Nov 02 23:36:55.385595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.385561 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q"] Nov 02 23:36:55.432846 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:55.432818 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-0fac41-openshift-default-77f89dd4f-n749q"] Nov 02 23:36:56.022774 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:36:56.022725 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:36:56.712976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:56.712933 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" path="/var/lib/kubelet/pods/0083ae5e-d949-47ce-8716-fd952f06f490/volumes" Nov 02 23:36:57.448050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:57.448020 5672 patch_prober.go:29] interesting pod/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:36:57.448241 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:57.448074 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:36:59.267787 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.267756 5672 generic.go:356] "Generic (PLEG): container finished" podID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerID="2754069ce4878620abcf94b60cbd83d89c43207d85be35d7024f4eb2261671b6" exitCode=0 Nov 02 23:36:59.268171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.267829 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" event={"ID":"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f","Type":"ContainerDied","Data":"2754069ce4878620abcf94b60cbd83d89c43207d85be35d7024f4eb2261671b6"} Nov 02 23:36:59.268171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.267899 5672 scope.go:122] "RemoveContainer" containerID="f12c43a7c27fb401fa0cfc1a62126cc5c8dc8365281ccdca6bf71f6c831c95c5" Nov 02 23:36:59.292033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.291992 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:36:59.404941 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.404905 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-envoy\") pod \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " Nov 02 23:36:59.405127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.404958 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-socket\") pod \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " Nov 02 23:36:59.405127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.405000 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-certs\") pod \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " Nov 02 23:36:59.405127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.405030 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-podinfo\") pod \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " Nov 02 23:36:59.405330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.405138 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-credential-socket\") pod \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " Nov 02 23:36:59.405330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.405218 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istiod-ca-cert\") pod \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " Nov 02 23:36:59.405330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.405245 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-token\" (UniqueName: \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-token\") pod \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " Nov 02 23:36:59.405330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.405278 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-data\") pod \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " Nov 02 23:36:59.405522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.405350 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-kube-api-access-d7mch\" (UniqueName: \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-kube-api-access-d7mch\") pod \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\" (UID: \"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f\") " Nov 02 23:36:59.406029 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.405832 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-credential-socket" pod "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" (UID: "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:59.406385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.406198 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-socket" pod "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" (UID: "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:59.407645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.407365 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istiod-ca-cert" pod "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" (UID: "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:36:59.407645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.407607 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-data" pod "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" (UID: "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:59.407861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.407835 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-certs" pod "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" (UID: "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:59.408460 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.408426 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-envoy" pod "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" (UID: "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:36:59.411084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.411051 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-podinfo" pod "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" (UID: "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:36:59.411220 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.411166 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-token" pod "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" (UID: "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:59.414296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.414269 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-kube-api-access-d7mch" pod "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" (UID: "bd8bd222-7c29-4a78-8ab3-a83cb9ed076f"). InnerVolumeSpecName "kube-api-access-d7mch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:36:59.506593 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.506560 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7mch\" (UniqueName: \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-kube-api-access-d7mch\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:59.506593 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.506589 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:59.506593 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.506598 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:59.506825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.506607 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:59.506825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.506616 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:59.506825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.506625 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:59.506825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.506641 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:59.506825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.506649 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:36:59.506825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:36:59.506658 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:00.277776 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:00.277743 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" Nov 02 23:37:00.278295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:00.277784 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj" event={"ID":"bd8bd222-7c29-4a78-8ab3-a83cb9ed076f","Type":"ContainerDied","Data":"b9dec5228c9f1d505f9b9c3bbb0a36f8b77bf5d5d74757ce0b2a00be91dd63e0"} Nov 02 23:37:00.278295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:00.277841 5672 scope.go:122] "RemoveContainer" containerID="2754069ce4878620abcf94b60cbd83d89c43207d85be35d7024f4eb2261671b6" Nov 02 23:37:00.539276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:00.539183 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj"] Nov 02 23:37:00.542593 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:00.542554 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-1c14e6-openshift-default-6c88454d8f-879fj"] Nov 02 23:37:00.712655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:00.712625 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" path="/var/lib/kubelet/pods/bd8bd222-7c29-4a78-8ab3-a83cb9ed076f/volumes" Nov 02 23:37:03.219472 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219439 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c"] Nov 02 23:37:03.219867 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219839 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" Nov 02 23:37:03.219867 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219850 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" Nov 02 23:37:03.219867 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219865 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" Nov 02 23:37:03.219970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219871 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" Nov 02 23:37:03.219970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219880 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" Nov 02 23:37:03.219970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219886 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" Nov 02 23:37:03.219970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219896 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" Nov 02 23:37:03.219970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219901 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" Nov 02 23:37:03.219970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219921 5672 cpu_manager.go:414] "containerMap: removing container" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" Nov 02 23:37:03.219970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219926 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" Nov 02 23:37:03.219970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219941 5672 cpu_manager.go:414] "containerMap: removing container" podUID="39f86f02-c5ac-45a7-b39f-2a13973ac864" containerName="revoke-keys" Nov 02 23:37:03.219970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219948 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f86f02-c5ac-45a7-b39f-2a13973ac864" containerName="revoke-keys" Nov 02 23:37:03.219970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219963 5672 cpu_manager.go:414] "containerMap: removing container" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" Nov 02 23:37:03.219970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.219968 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" Nov 02 23:37:03.220322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.220042 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" Nov 02 23:37:03.220322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.220052 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f86f02-c5ac-45a7-b39f-2a13973ac864" containerName="revoke-keys" Nov 02 23:37:03.220322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.220060 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="70176952-4b2e-4939-98cd-62d60951971d" containerName="istio-proxy" Nov 02 23:37:03.220322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.220068 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" Nov 02 23:37:03.220322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.220073 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0083ae5e-d949-47ce-8716-fd952f06f490" containerName="istio-proxy" Nov 02 23:37:03.220322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.220079 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" Nov 02 23:37:03.220322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.220086 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8bd222-7c29-4a78-8ab3-a83cb9ed076f" containerName="istio-proxy" Nov 02 23:37:03.224718 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.224699 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:03.276863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.276826 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c"] Nov 02 23:37:03.320112 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.320085 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:37:03.320420 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.320395 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:37:03.320501 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.320401 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:37:03.320961 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.320937 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:37:03.348899 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.348866 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30fb1af1-drq6c\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:03.349037 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.348931 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30fb1af1-drq6c\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:03.349037 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.349020 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgn6\" (UniqueName: \"kubernetes.io/projected/1c207c04-3728-4bf2-98f6-091f61976ba4-kube-api-access-nkgn6\") pod \"maas-api-revoke-sub-30fb1af1-drq6c\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:03.450173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.450121 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30fb1af1-drq6c\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:03.450372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.450196 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30fb1af1-drq6c\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:03.450372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.450231 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgn6\" (UniqueName: \"kubernetes.io/projected/1c207c04-3728-4bf2-98f6-091f61976ba4-kube-api-access-nkgn6\") pod \"maas-api-revoke-sub-30fb1af1-drq6c\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:03.450897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.450868 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30fb1af1-drq6c\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:03.452664 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.452631 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30fb1af1-drq6c\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:03.503189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.503110 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgn6\" (UniqueName: \"kubernetes.io/projected/1c207c04-3728-4bf2-98f6-091f61976ba4-kube-api-access-nkgn6\") pod \"maas-api-revoke-sub-30fb1af1-drq6c\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:03.538086 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.538059 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:03.909572 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:03.909545 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c"] Nov 02 23:37:03.910524 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:37:03.910498 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c207c04_3728_4bf2_98f6_091f61976ba4.slice/crio-1d1806d89fe2c9740482c3dd7f250858772c3e555b4eb3175c10838a2a95cb9c WatchSource:0}: Error finding container 1d1806d89fe2c9740482c3dd7f250858772c3e555b4eb3175c10838a2a95cb9c: Status 404 returned error can't find the container with id 1d1806d89fe2c9740482c3dd7f250858772c3e555b4eb3175c10838a2a95cb9c Nov 02 23:37:04.305751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:04.305712 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" event={"ID":"1c207c04-3728-4bf2-98f6-091f61976ba4","Type":"ContainerStarted","Data":"fe3154c7de6f3cafd355d357e58590ead3451edac1331d7ef0816f629cdf4d33"} Nov 02 23:37:04.306101 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:04.305770 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" event={"ID":"1c207c04-3728-4bf2-98f6-091f61976ba4","Type":"ContainerStarted","Data":"1d1806d89fe2c9740482c3dd7f250858772c3e555b4eb3175c10838a2a95cb9c"} Nov 02 23:37:04.416570 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:04.416520 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" podStartSLOduration=1.416506489 podStartE2EDuration="1.416506489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:04.415077014 +0000 UTC m=+2617.822388843" watchObservedRunningTime="2026-11-02 23:37:04.416506489 +0000 UTC m=+2617.823818371" Nov 02 23:37:05.314813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.314778 5672 generic.go:356] "Generic (PLEG): container finished" podID="1c207c04-3728-4bf2-98f6-091f61976ba4" containerID="fe3154c7de6f3cafd355d357e58590ead3451edac1331d7ef0816f629cdf4d33" exitCode=0 Nov 02 23:37:05.315267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.314870 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" event={"ID":"1c207c04-3728-4bf2-98f6-091f61976ba4","Type":"ContainerDied","Data":"fe3154c7de6f3cafd355d357e58590ead3451edac1331d7ef0816f629cdf4d33"} Nov 02 23:37:05.611753 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.611644 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6"] Nov 02 23:37:05.615792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.615769 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:05.668632 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.668602 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6"] Nov 02 23:37:05.773666 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.773624 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4447eb-vllt6\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:05.773889 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.773869 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mr9\" (UniqueName: \"kubernetes.io/projected/1ca7a002-ac54-4415-8fd2-b923419dcfa4-kube-api-access-t5mr9\") pod \"maas-api-revoke-sub-ac4447eb-vllt6\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:05.773941 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.773906 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4447eb-vllt6\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:05.875511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.875431 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mr9\" (UniqueName: \"kubernetes.io/projected/1ca7a002-ac54-4415-8fd2-b923419dcfa4-kube-api-access-t5mr9\") pod \"maas-api-revoke-sub-ac4447eb-vllt6\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:05.875511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.875468 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4447eb-vllt6\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:05.875511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.875508 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4447eb-vllt6\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:05.876120 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.876090 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4447eb-vllt6\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:05.877831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.877810 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4447eb-vllt6\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:05.929692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:05.929672 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mr9\" (UniqueName: \"kubernetes.io/projected/1ca7a002-ac54-4415-8fd2-b923419dcfa4-kube-api-access-t5mr9\") pod \"maas-api-revoke-sub-ac4447eb-vllt6\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:06.229157 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.229113 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:06.482625 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.482564 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:06.582711 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.582670 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c207c04-3728-4bf2-98f6-091f61976ba4-kube-api-access-nkgn6\" (UniqueName: \"kubernetes.io/projected/1c207c04-3728-4bf2-98f6-091f61976ba4-kube-api-access-nkgn6\") pod \"1c207c04-3728-4bf2-98f6-091f61976ba4\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " Nov 02 23:37:06.582896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.582717 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-openshift-ca\") pod \"1c207c04-3728-4bf2-98f6-091f61976ba4\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " Nov 02 23:37:06.582953 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.582894 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-xks-ca\") pod \"1c207c04-3728-4bf2-98f6-091f61976ba4\" (UID: \"1c207c04-3728-4bf2-98f6-091f61976ba4\") " Nov 02 23:37:06.583111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.583087 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-openshift-ca" pod "1c207c04-3728-4bf2-98f6-091f61976ba4" (UID: "1c207c04-3728-4bf2-98f6-091f61976ba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:06.583287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.583270 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:06.584858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.584830 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-xks-ca" pod "1c207c04-3728-4bf2-98f6-091f61976ba4" (UID: "1c207c04-3728-4bf2-98f6-091f61976ba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:37:06.584858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.584853 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c207c04-3728-4bf2-98f6-091f61976ba4-kube-api-access-nkgn6" pod "1c207c04-3728-4bf2-98f6-091f61976ba4" (UID: "1c207c04-3728-4bf2-98f6-091f61976ba4"). InnerVolumeSpecName "kube-api-access-nkgn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:06.607217 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:37:06.607191 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca7a002_ac54_4415_8fd2_b923419dcfa4.slice/crio-7000baa6006e99554e60eb632658fe1dadee7f3badaa39d65b7a84e739822cae WatchSource:0}: Error finding container 7000baa6006e99554e60eb632658fe1dadee7f3badaa39d65b7a84e739822cae: Status 404 returned error can't find the container with id 7000baa6006e99554e60eb632658fe1dadee7f3badaa39d65b7a84e739822cae Nov 02 23:37:06.639097 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.639054 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6"] Nov 02 23:37:06.683965 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.683930 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c207c04-3728-4bf2-98f6-091f61976ba4-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:06.683965 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:06.683961 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgn6\" (UniqueName: \"kubernetes.io/projected/1c207c04-3728-4bf2-98f6-091f61976ba4-kube-api-access-nkgn6\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:07.330064 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:07.330030 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" Nov 02 23:37:07.330287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:07.330059 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c" event={"ID":"1c207c04-3728-4bf2-98f6-091f61976ba4","Type":"ContainerDied","Data":"1d1806d89fe2c9740482c3dd7f250858772c3e555b4eb3175c10838a2a95cb9c"} Nov 02 23:37:07.330287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:07.330115 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d1806d89fe2c9740482c3dd7f250858772c3e555b4eb3175c10838a2a95cb9c" Nov 02 23:37:07.332382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:07.332352 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" event={"ID":"1ca7a002-ac54-4415-8fd2-b923419dcfa4","Type":"ContainerStarted","Data":"e19f3aa7394db68408af2749796564ce267192adf3828d2b8aa8768c5cdcd965"} Nov 02 23:37:07.332530 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:07.332404 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" event={"ID":"1ca7a002-ac54-4415-8fd2-b923419dcfa4","Type":"ContainerStarted","Data":"7000baa6006e99554e60eb632658fe1dadee7f3badaa39d65b7a84e739822cae"} Nov 02 23:37:07.442930 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:07.442859 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" podStartSLOduration=2.442841889 podStartE2EDuration="2.442841889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:07.439597757 +0000 UTC m=+2620.846909587" watchObservedRunningTime="2026-11-02 23:37:07.442841889 +0000 UTC m=+2620.850153719" Nov 02 23:37:08.341640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:08.341600 5672 generic.go:356] "Generic (PLEG): container finished" podID="1ca7a002-ac54-4415-8fd2-b923419dcfa4" containerID="e19f3aa7394db68408af2749796564ce267192adf3828d2b8aa8768c5cdcd965" exitCode=0 Nov 02 23:37:08.342109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:08.341668 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" event={"ID":"1ca7a002-ac54-4415-8fd2-b923419dcfa4","Type":"ContainerDied","Data":"e19f3aa7394db68408af2749796564ce267192adf3828d2b8aa8768c5cdcd965"} Nov 02 23:37:09.508893 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:09.508869 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:09.612890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:09.612817 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-xks-ca\") pod \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " Nov 02 23:37:09.613021 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:09.612930 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-openshift-ca\") pod \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " Nov 02 23:37:09.613021 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:09.613005 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca7a002-ac54-4415-8fd2-b923419dcfa4-kube-api-access-t5mr9\" (UniqueName: \"kubernetes.io/projected/1ca7a002-ac54-4415-8fd2-b923419dcfa4-kube-api-access-t5mr9\") pod \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\" (UID: \"1ca7a002-ac54-4415-8fd2-b923419dcfa4\") " Nov 02 23:37:09.613308 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:09.613283 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-openshift-ca" pod "1ca7a002-ac54-4415-8fd2-b923419dcfa4" (UID: "1ca7a002-ac54-4415-8fd2-b923419dcfa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:09.613425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:09.613393 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:09.615196 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:09.615165 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-xks-ca" pod "1ca7a002-ac54-4415-8fd2-b923419dcfa4" (UID: "1ca7a002-ac54-4415-8fd2-b923419dcfa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:37:09.616700 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:09.616674 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca7a002-ac54-4415-8fd2-b923419dcfa4-kube-api-access-t5mr9" pod "1ca7a002-ac54-4415-8fd2-b923419dcfa4" (UID: "1ca7a002-ac54-4415-8fd2-b923419dcfa4"). InnerVolumeSpecName "kube-api-access-t5mr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:09.714225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:09.714191 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mr9\" (UniqueName: \"kubernetes.io/projected/1ca7a002-ac54-4415-8fd2-b923419dcfa4-kube-api-access-t5mr9\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:09.714225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:09.714219 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca7a002-ac54-4415-8fd2-b923419dcfa4-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:10.356093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:10.356061 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" Nov 02 23:37:10.356317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:10.356096 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6" event={"ID":"1ca7a002-ac54-4415-8fd2-b923419dcfa4","Type":"ContainerDied","Data":"7000baa6006e99554e60eb632658fe1dadee7f3badaa39d65b7a84e739822cae"} Nov 02 23:37:10.356317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:10.356168 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7000baa6006e99554e60eb632658fe1dadee7f3badaa39d65b7a84e739822cae" Nov 02 23:37:11.534535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:11.534499 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:37:14.594549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.594517 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx"] Nov 02 23:37:14.594967 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.594953 5672 cpu_manager.go:414] "containerMap: removing container" podUID="1ca7a002-ac54-4415-8fd2-b923419dcfa4" containerName="revoke-keys" Nov 02 23:37:14.595007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.594968 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca7a002-ac54-4415-8fd2-b923419dcfa4" containerName="revoke-keys" Nov 02 23:37:14.595007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.594987 5672 cpu_manager.go:414] "containerMap: removing container" podUID="1c207c04-3728-4bf2-98f6-091f61976ba4" containerName="revoke-keys" Nov 02 23:37:14.595007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.594992 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c207c04-3728-4bf2-98f6-091f61976ba4" containerName="revoke-keys" Nov 02 23:37:14.595094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.595072 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca7a002-ac54-4415-8fd2-b923419dcfa4" containerName="revoke-keys" Nov 02 23:37:14.595094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.595082 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c207c04-3728-4bf2-98f6-091f61976ba4" containerName="revoke-keys" Nov 02 23:37:14.598581 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.598561 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:14.650756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.650719 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx"] Nov 02 23:37:14.693302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.693274 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:37:14.693302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.693299 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:37:14.693757 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.693740 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:37:14.693793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.693763 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:37:14.712946 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.712921 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59900d47-d15a-493a-800b-88bead4da834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56a829f7-mjrwx\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:14.713107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.713055 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59900d47-d15a-493a-800b-88bead4da834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56a829f7-mjrwx\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:14.713213 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.713197 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjsjb\" (UniqueName: \"kubernetes.io/projected/59900d47-d15a-493a-800b-88bead4da834-kube-api-access-jjsjb\") pod \"maas-api-revoke-sub-56a829f7-mjrwx\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:14.814838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.814805 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59900d47-d15a-493a-800b-88bead4da834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56a829f7-mjrwx\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:14.815027 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.814960 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjsjb\" (UniqueName: \"kubernetes.io/projected/59900d47-d15a-493a-800b-88bead4da834-kube-api-access-jjsjb\") pod \"maas-api-revoke-sub-56a829f7-mjrwx\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:14.815085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.815028 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59900d47-d15a-493a-800b-88bead4da834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56a829f7-mjrwx\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:14.815727 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.815702 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59900d47-d15a-493a-800b-88bead4da834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56a829f7-mjrwx\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:14.817268 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.817245 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59900d47-d15a-493a-800b-88bead4da834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56a829f7-mjrwx\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:14.868599 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.868530 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjsjb\" (UniqueName: \"kubernetes.io/projected/59900d47-d15a-493a-800b-88bead4da834-kube-api-access-jjsjb\") pod \"maas-api-revoke-sub-56a829f7-mjrwx\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:14.912441 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:14.912411 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:15.285308 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:15.285281 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx"] Nov 02 23:37:15.286278 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:37:15.286246 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59900d47_d15a_493a_800b_88bead4da834.slice/crio-f914416fa78f9362a9dcbe862b8520ab8f32452f34785d28d417e27de5082201 WatchSource:0}: Error finding container f914416fa78f9362a9dcbe862b8520ab8f32452f34785d28d417e27de5082201: Status 404 returned error can't find the container with id f914416fa78f9362a9dcbe862b8520ab8f32452f34785d28d417e27de5082201 Nov 02 23:37:15.396412 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:15.396375 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" event={"ID":"59900d47-d15a-493a-800b-88bead4da834","Type":"ContainerStarted","Data":"f914416fa78f9362a9dcbe862b8520ab8f32452f34785d28d417e27de5082201"} Nov 02 23:37:16.405574 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:16.405470 5672 generic.go:356] "Generic (PLEG): container finished" podID="59900d47-d15a-493a-800b-88bead4da834" containerID="afb21df32bbb75c3e28419292bfda2035fcef03e00cb482f1b163ec134e2731c" exitCode=0 Nov 02 23:37:16.405574 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:16.405551 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" event={"ID":"59900d47-d15a-493a-800b-88bead4da834","Type":"ContainerDied","Data":"afb21df32bbb75c3e28419292bfda2035fcef03e00cb482f1b163ec134e2731c"} Nov 02 23:37:17.566897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:17.566871 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:17.647038 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:17.646998 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59900d47-d15a-493a-800b-88bead4da834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59900d47-d15a-493a-800b-88bead4da834-maas-api-xks-ca\") pod \"59900d47-d15a-493a-800b-88bead4da834\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " Nov 02 23:37:17.647254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:17.647055 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59900d47-d15a-493a-800b-88bead4da834-kube-api-access-jjsjb\" (UniqueName: \"kubernetes.io/projected/59900d47-d15a-493a-800b-88bead4da834-kube-api-access-jjsjb\") pod \"59900d47-d15a-493a-800b-88bead4da834\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " Nov 02 23:37:17.647319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:17.647277 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59900d47-d15a-493a-800b-88bead4da834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59900d47-d15a-493a-800b-88bead4da834-maas-api-openshift-ca\") pod \"59900d47-d15a-493a-800b-88bead4da834\" (UID: \"59900d47-d15a-493a-800b-88bead4da834\") " Nov 02 23:37:17.647648 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:17.647619 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59900d47-d15a-493a-800b-88bead4da834-maas-api-openshift-ca" pod "59900d47-d15a-493a-800b-88bead4da834" (UID: "59900d47-d15a-493a-800b-88bead4da834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:17.649313 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:17.649290 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59900d47-d15a-493a-800b-88bead4da834-kube-api-access-jjsjb" pod "59900d47-d15a-493a-800b-88bead4da834" (UID: "59900d47-d15a-493a-800b-88bead4da834"). InnerVolumeSpecName "kube-api-access-jjsjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:17.649313 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:17.649294 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59900d47-d15a-493a-800b-88bead4da834-maas-api-xks-ca" pod "59900d47-d15a-493a-800b-88bead4da834" (UID: "59900d47-d15a-493a-800b-88bead4da834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:37:17.748715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:17.748666 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59900d47-d15a-493a-800b-88bead4da834-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:17.748715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:17.748709 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59900d47-d15a-493a-800b-88bead4da834-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:17.748715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:17.748724 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjsjb\" (UniqueName: \"kubernetes.io/projected/59900d47-d15a-493a-800b-88bead4da834-kube-api-access-jjsjb\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:18.420562 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:18.420524 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" event={"ID":"59900d47-d15a-493a-800b-88bead4da834","Type":"ContainerDied","Data":"f914416fa78f9362a9dcbe862b8520ab8f32452f34785d28d417e27de5082201"} Nov 02 23:37:18.420741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:18.420565 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx" Nov 02 23:37:18.420741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:18.420578 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f914416fa78f9362a9dcbe862b8520ab8f32452f34785d28d417e27de5082201" Nov 02 23:37:24.530792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.530748 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg"] Nov 02 23:37:24.531481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.531460 5672 cpu_manager.go:414] "containerMap: removing container" podUID="59900d47-d15a-493a-800b-88bead4da834" containerName="revoke-keys" Nov 02 23:37:24.531568 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.531484 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59900d47-d15a-493a-800b-88bead4da834" containerName="revoke-keys" Nov 02 23:37:24.531626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.531604 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="59900d47-d15a-493a-800b-88bead4da834" containerName="revoke-keys" Nov 02 23:37:24.534963 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.534940 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:24.588532 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.588501 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg"] Nov 02 23:37:24.622452 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.622416 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8mp\" (UniqueName: \"kubernetes.io/projected/99bbc390-437d-4e6c-acf5-2853dc335d03-kube-api-access-lq8mp\") pod \"maas-api-revoke-sub-5293a438-fx7wg\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:24.622452 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.622456 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5293a438-fx7wg\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:24.622709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.622598 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5293a438-fx7wg\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:24.630775 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.630751 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:37:24.630922 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.630750 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:37:24.630922 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.630750 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:37:24.630922 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.630826 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:37:24.724074 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.724043 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8mp\" (UniqueName: \"kubernetes.io/projected/99bbc390-437d-4e6c-acf5-2853dc335d03-kube-api-access-lq8mp\") pod \"maas-api-revoke-sub-5293a438-fx7wg\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:24.724259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.724080 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5293a438-fx7wg\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:24.724259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.724127 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5293a438-fx7wg\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:24.724755 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.724728 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5293a438-fx7wg\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:24.726751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.726724 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5293a438-fx7wg\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:24.777758 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.777720 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8mp\" (UniqueName: \"kubernetes.io/projected/99bbc390-437d-4e6c-acf5-2853dc335d03-kube-api-access-lq8mp\") pod \"maas-api-revoke-sub-5293a438-fx7wg\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:24.849247 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:24.849157 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:25.216579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:25.216550 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg"] Nov 02 23:37:25.218353 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:37:25.218317 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bbc390_437d_4e6c_acf5_2853dc335d03.slice/crio-993be8f78863242a08cde48148de3be1ce44aed9943d0ae2d4d934aa4e09fbe7 WatchSource:0}: Error finding container 993be8f78863242a08cde48148de3be1ce44aed9943d0ae2d4d934aa4e09fbe7: Status 404 returned error can't find the container with id 993be8f78863242a08cde48148de3be1ce44aed9943d0ae2d4d934aa4e09fbe7 Nov 02 23:37:25.469207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:25.469173 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" event={"ID":"99bbc390-437d-4e6c-acf5-2853dc335d03","Type":"ContainerStarted","Data":"993be8f78863242a08cde48148de3be1ce44aed9943d0ae2d4d934aa4e09fbe7"} Nov 02 23:37:26.021213 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:37:26.021173 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:37:26.478204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:26.478168 5672 generic.go:356] "Generic (PLEG): container finished" podID="99bbc390-437d-4e6c-acf5-2853dc335d03" containerID="643d7e226c2452e8bc4d37242ed2fc10e077f9293645541189916cddd4540300" exitCode=0 Nov 02 23:37:26.478393 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:26.478251 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" event={"ID":"99bbc390-437d-4e6c-acf5-2853dc335d03","Type":"ContainerDied","Data":"643d7e226c2452e8bc4d37242ed2fc10e077f9293645541189916cddd4540300"} Nov 02 23:37:27.640090 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:27.640060 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:27.756912 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:27.756824 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-openshift-ca\") pod \"99bbc390-437d-4e6c-acf5-2853dc335d03\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " Nov 02 23:37:27.757081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:27.756915 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99bbc390-437d-4e6c-acf5-2853dc335d03-kube-api-access-lq8mp\" (UniqueName: \"kubernetes.io/projected/99bbc390-437d-4e6c-acf5-2853dc335d03-kube-api-access-lq8mp\") pod \"99bbc390-437d-4e6c-acf5-2853dc335d03\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " Nov 02 23:37:27.757081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:27.757001 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-xks-ca\") pod \"99bbc390-437d-4e6c-acf5-2853dc335d03\" (UID: \"99bbc390-437d-4e6c-acf5-2853dc335d03\") " Nov 02 23:37:27.757276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:27.757249 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-openshift-ca" pod "99bbc390-437d-4e6c-acf5-2853dc335d03" (UID: "99bbc390-437d-4e6c-acf5-2853dc335d03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:27.757483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:27.757463 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:27.759302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:27.759277 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-xks-ca" pod "99bbc390-437d-4e6c-acf5-2853dc335d03" (UID: "99bbc390-437d-4e6c-acf5-2853dc335d03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:37:27.759585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:27.759562 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bbc390-437d-4e6c-acf5-2853dc335d03-kube-api-access-lq8mp" pod "99bbc390-437d-4e6c-acf5-2853dc335d03" (UID: "99bbc390-437d-4e6c-acf5-2853dc335d03"). InnerVolumeSpecName "kube-api-access-lq8mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:27.858184 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:27.858129 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq8mp\" (UniqueName: \"kubernetes.io/projected/99bbc390-437d-4e6c-acf5-2853dc335d03-kube-api-access-lq8mp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:27.858184 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:27.858178 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bbc390-437d-4e6c-acf5-2853dc335d03-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:28.500947 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.500914 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" event={"ID":"99bbc390-437d-4e6c-acf5-2853dc335d03","Type":"ContainerDied","Data":"993be8f78863242a08cde48148de3be1ce44aed9943d0ae2d4d934aa4e09fbe7"} Nov 02 23:37:28.501120 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.500968 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="993be8f78863242a08cde48148de3be1ce44aed9943d0ae2d4d934aa4e09fbe7" Nov 02 23:37:28.501120 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.500922 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg" Nov 02 23:37:28.680342 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.680269 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4"] Nov 02 23:37:28.680753 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.680738 5672 cpu_manager.go:414] "containerMap: removing container" podUID="99bbc390-437d-4e6c-acf5-2853dc335d03" containerName="revoke-keys" Nov 02 23:37:28.680797 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.680754 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bbc390-437d-4e6c-acf5-2853dc335d03" containerName="revoke-keys" Nov 02 23:37:28.680852 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.680842 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bbc390-437d-4e6c-acf5-2853dc335d03" containerName="revoke-keys" Nov 02 23:37:28.685507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.685487 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:28.734612 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.734581 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:37:28.734915 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.734889 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:37:28.734915 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.734902 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:37:28.735099 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.734927 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:37:28.736311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.736291 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4"] Nov 02 23:37:28.769320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.769292 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d399be-btjm4\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:28.769484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.769342 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpx7s\" (UniqueName: \"kubernetes.io/projected/989cbadc-d89d-4c78-859d-fa0f0eecaa31-kube-api-access-mpx7s\") pod \"maas-api-revoke-sub-09d399be-btjm4\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:28.769484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.769365 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d399be-btjm4\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:28.871326 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.871261 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d399be-btjm4\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:28.873176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.871633 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpx7s\" (UniqueName: \"kubernetes.io/projected/989cbadc-d89d-4c78-859d-fa0f0eecaa31-kube-api-access-mpx7s\") pod \"maas-api-revoke-sub-09d399be-btjm4\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:28.873176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.871688 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d399be-btjm4\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:28.873176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.872992 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09d399be-btjm4\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:28.878055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.875123 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09d399be-btjm4\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:28.925419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.925387 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpx7s\" (UniqueName: \"kubernetes.io/projected/989cbadc-d89d-4c78-859d-fa0f0eecaa31-kube-api-access-mpx7s\") pod \"maas-api-revoke-sub-09d399be-btjm4\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:28.997835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:28.997792 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:29.077771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.077740 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs"] Nov 02 23:37:29.082397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.082370 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:29.134251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.134215 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs"] Nov 02 23:37:29.176362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.176332 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4a3413-m25rs\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:29.176532 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.176376 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4a3413-m25rs\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:29.176532 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.176487 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzfb6\" (UniqueName: \"kubernetes.io/projected/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-kube-api-access-fzfb6\") pod \"maas-api-revoke-sub-8d4a3413-m25rs\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:29.277976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.277891 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzfb6\" (UniqueName: \"kubernetes.io/projected/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-kube-api-access-fzfb6\") pod \"maas-api-revoke-sub-8d4a3413-m25rs\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:29.278176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.278020 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4a3413-m25rs\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:29.278176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.278049 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4a3413-m25rs\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:29.278893 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.278862 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4a3413-m25rs\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:29.280401 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.280380 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4a3413-m25rs\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:29.331076 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.331050 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzfb6\" (UniqueName: \"kubernetes.io/projected/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-kube-api-access-fzfb6\") pod \"maas-api-revoke-sub-8d4a3413-m25rs\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:29.366661 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.366632 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4"] Nov 02 23:37:29.366763 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:37:29.366710 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989cbadc_d89d_4c78_859d_fa0f0eecaa31.slice/crio-4b0d475d23fcf37d2230ef486ae385f6b7c021d438c5058dc3cc9d6133a3bc03 WatchSource:0}: Error finding container 4b0d475d23fcf37d2230ef486ae385f6b7c021d438c5058dc3cc9d6133a3bc03: Status 404 returned error can't find the container with id 4b0d475d23fcf37d2230ef486ae385f6b7c021d438c5058dc3cc9d6133a3bc03 Nov 02 23:37:29.397809 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.397780 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:29.510110 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.510071 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" event={"ID":"989cbadc-d89d-4c78-859d-fa0f0eecaa31","Type":"ContainerStarted","Data":"4b0d475d23fcf37d2230ef486ae385f6b7c021d438c5058dc3cc9d6133a3bc03"} Nov 02 23:37:29.777708 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:29.777671 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs"] Nov 02 23:37:29.778909 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:37:29.778883 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbeaaa0_4be3_4be2_afa2_87604003d7f8.slice/crio-3e3afb6b5e2f52a4b93887911f4c9a7f5edb1c5671e70747c24f2d03d45616c1 WatchSource:0}: Error finding container 3e3afb6b5e2f52a4b93887911f4c9a7f5edb1c5671e70747c24f2d03d45616c1: Status 404 returned error can't find the container with id 3e3afb6b5e2f52a4b93887911f4c9a7f5edb1c5671e70747c24f2d03d45616c1 Nov 02 23:37:30.518704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:30.518605 5672 generic.go:356] "Generic (PLEG): container finished" podID="989cbadc-d89d-4c78-859d-fa0f0eecaa31" containerID="1df5a82a5b7ad7e11506921e7a3bf13c7ba5828c322347ebbf2bd02b52cf6d7b" exitCode=0 Nov 02 23:37:30.518704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:30.518683 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" event={"ID":"989cbadc-d89d-4c78-859d-fa0f0eecaa31","Type":"ContainerDied","Data":"1df5a82a5b7ad7e11506921e7a3bf13c7ba5828c322347ebbf2bd02b52cf6d7b"} Nov 02 23:37:30.520536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:30.520513 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" event={"ID":"2dbeaaa0-4be3-4be2-afa2-87604003d7f8","Type":"ContainerStarted","Data":"1481614ffe55b1e108b46e96f80a002d9a18c7fbd7a78238e2936558f6791fca"} Nov 02 23:37:30.520662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:30.520556 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" event={"ID":"2dbeaaa0-4be3-4be2-afa2-87604003d7f8","Type":"ContainerStarted","Data":"3e3afb6b5e2f52a4b93887911f4c9a7f5edb1c5671e70747c24f2d03d45616c1"} Nov 02 23:37:30.733143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:30.733081 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" podStartSLOduration=1.733063264 podStartE2EDuration="1.733063264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:30.730674861 +0000 UTC m=+2644.137986681" watchObservedRunningTime="2026-11-02 23:37:30.733063264 +0000 UTC m=+2644.140375097" Nov 02 23:37:31.529383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.529350 5672 generic.go:356] "Generic (PLEG): container finished" podID="2dbeaaa0-4be3-4be2-afa2-87604003d7f8" containerID="1481614ffe55b1e108b46e96f80a002d9a18c7fbd7a78238e2936558f6791fca" exitCode=0 Nov 02 23:37:31.529846 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.529425 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" event={"ID":"2dbeaaa0-4be3-4be2-afa2-87604003d7f8","Type":"ContainerDied","Data":"1481614ffe55b1e108b46e96f80a002d9a18c7fbd7a78238e2936558f6791fca"} Nov 02 23:37:31.689169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.689129 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:31.803237 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.803115 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-openshift-ca\") pod \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " Nov 02 23:37:31.803438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.803256 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989cbadc-d89d-4c78-859d-fa0f0eecaa31-kube-api-access-mpx7s\" (UniqueName: \"kubernetes.io/projected/989cbadc-d89d-4c78-859d-fa0f0eecaa31-kube-api-access-mpx7s\") pod \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " Nov 02 23:37:31.803438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.803319 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-xks-ca\") pod \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\" (UID: \"989cbadc-d89d-4c78-859d-fa0f0eecaa31\") " Nov 02 23:37:31.803586 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.803517 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-openshift-ca" pod "989cbadc-d89d-4c78-859d-fa0f0eecaa31" (UID: "989cbadc-d89d-4c78-859d-fa0f0eecaa31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:31.803831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.803794 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:31.806337 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.806303 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989cbadc-d89d-4c78-859d-fa0f0eecaa31-kube-api-access-mpx7s" pod "989cbadc-d89d-4c78-859d-fa0f0eecaa31" (UID: "989cbadc-d89d-4c78-859d-fa0f0eecaa31"). InnerVolumeSpecName "kube-api-access-mpx7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:31.815055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.814994 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-xks-ca" pod "989cbadc-d89d-4c78-859d-fa0f0eecaa31" (UID: "989cbadc-d89d-4c78-859d-fa0f0eecaa31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:37:31.904381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.904341 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpx7s\" (UniqueName: \"kubernetes.io/projected/989cbadc-d89d-4c78-859d-fa0f0eecaa31-kube-api-access-mpx7s\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:31.904381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:31.904371 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989cbadc-d89d-4c78-859d-fa0f0eecaa31-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:32.539413 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.539362 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" Nov 02 23:37:32.539413 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.539396 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4" event={"ID":"989cbadc-d89d-4c78-859d-fa0f0eecaa31","Type":"ContainerDied","Data":"4b0d475d23fcf37d2230ef486ae385f6b7c021d438c5058dc3cc9d6133a3bc03"} Nov 02 23:37:32.539989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.539456 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b0d475d23fcf37d2230ef486ae385f6b7c021d438c5058dc3cc9d6133a3bc03" Nov 02 23:37:32.696665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.696637 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:32.814620 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.814531 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-openshift-ca\") pod \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " Nov 02 23:37:32.814794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.814638 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-kube-api-access-fzfb6\" (UniqueName: \"kubernetes.io/projected/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-kube-api-access-fzfb6\") pod \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " Nov 02 23:37:32.814794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.814669 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-xks-ca\") pod \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\" (UID: \"2dbeaaa0-4be3-4be2-afa2-87604003d7f8\") " Nov 02 23:37:32.814990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.814967 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-openshift-ca" pod "2dbeaaa0-4be3-4be2-afa2-87604003d7f8" (UID: "2dbeaaa0-4be3-4be2-afa2-87604003d7f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:32.815136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.815117 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:32.817237 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.817208 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-kube-api-access-fzfb6" pod "2dbeaaa0-4be3-4be2-afa2-87604003d7f8" (UID: "2dbeaaa0-4be3-4be2-afa2-87604003d7f8"). InnerVolumeSpecName "kube-api-access-fzfb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:32.817391 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.817367 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-xks-ca" pod "2dbeaaa0-4be3-4be2-afa2-87604003d7f8" (UID: "2dbeaaa0-4be3-4be2-afa2-87604003d7f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:37:32.916660 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.916614 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzfb6\" (UniqueName: \"kubernetes.io/projected/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-kube-api-access-fzfb6\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:32.916660 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:32.916652 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dbeaaa0-4be3-4be2-afa2-87604003d7f8-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:33.550224 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:33.550191 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" Nov 02 23:37:33.550748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:33.550191 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs" event={"ID":"2dbeaaa0-4be3-4be2-afa2-87604003d7f8","Type":"ContainerDied","Data":"3e3afb6b5e2f52a4b93887911f4c9a7f5edb1c5671e70747c24f2d03d45616c1"} Nov 02 23:37:33.550748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:33.550339 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e3afb6b5e2f52a4b93887911f4c9a7f5edb1c5671e70747c24f2d03d45616c1" Nov 02 23:37:39.432482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.432438 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz"] Nov 02 23:37:39.433089 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.433069 5672 cpu_manager.go:414] "containerMap: removing container" podUID="989cbadc-d89d-4c78-859d-fa0f0eecaa31" containerName="revoke-keys" Nov 02 23:37:39.433192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.433092 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989cbadc-d89d-4c78-859d-fa0f0eecaa31" containerName="revoke-keys" Nov 02 23:37:39.433192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.433132 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2dbeaaa0-4be3-4be2-afa2-87604003d7f8" containerName="revoke-keys" Nov 02 23:37:39.433192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.433139 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dbeaaa0-4be3-4be2-afa2-87604003d7f8" containerName="revoke-keys" Nov 02 23:37:39.433370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.433285 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="989cbadc-d89d-4c78-859d-fa0f0eecaa31" containerName="revoke-keys" Nov 02 23:37:39.433370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.433301 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dbeaaa0-4be3-4be2-afa2-87604003d7f8" containerName="revoke-keys" Nov 02 23:37:39.439042 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.439019 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:39.487187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.487134 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz"] Nov 02 23:37:39.529760 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.529728 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:37:39.530014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.529735 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:37:39.536306 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.536280 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:37:39.536502 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.536281 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:37:39.544949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.544906 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt2js\" (UniqueName: \"kubernetes.io/projected/8d622289-ec5c-4d0a-a380-f5d077ed6b17-kube-api-access-jt2js\") pod \"maas-api-revoke-sub-42665ce8-m4jpz\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:39.545088 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.544958 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42665ce8-m4jpz\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:39.545261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.545230 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42665ce8-m4jpz\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:39.648172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.646942 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42665ce8-m4jpz\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:39.648172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.647067 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt2js\" (UniqueName: \"kubernetes.io/projected/8d622289-ec5c-4d0a-a380-f5d077ed6b17-kube-api-access-jt2js\") pod \"maas-api-revoke-sub-42665ce8-m4jpz\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:39.648172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.647438 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42665ce8-m4jpz\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:39.648172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.648129 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42665ce8-m4jpz\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:39.665459 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.664125 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42665ce8-m4jpz\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:39.700613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.700545 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt2js\" (UniqueName: \"kubernetes.io/projected/8d622289-ec5c-4d0a-a380-f5d077ed6b17-kube-api-access-jt2js\") pod \"maas-api-revoke-sub-42665ce8-m4jpz\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:39.753269 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:39.753241 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:40.173800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:40.173771 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz"] Nov 02 23:37:40.175840 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:37:40.175814 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d622289_ec5c_4d0a_a380_f5d077ed6b17.slice/crio-ad4f8e80aa7cb6663aec54b08e76e81825ba41a66eae72f17b4c4a8a44450d54 WatchSource:0}: Error finding container ad4f8e80aa7cb6663aec54b08e76e81825ba41a66eae72f17b4c4a8a44450d54: Status 404 returned error can't find the container with id ad4f8e80aa7cb6663aec54b08e76e81825ba41a66eae72f17b4c4a8a44450d54 Nov 02 23:37:40.626744 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:40.626704 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" event={"ID":"8d622289-ec5c-4d0a-a380-f5d077ed6b17","Type":"ContainerStarted","Data":"ad4f8e80aa7cb6663aec54b08e76e81825ba41a66eae72f17b4c4a8a44450d54"} Nov 02 23:37:41.636742 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:41.636710 5672 generic.go:356] "Generic (PLEG): container finished" podID="8d622289-ec5c-4d0a-a380-f5d077ed6b17" containerID="b85dbe0a26ea3442ad375622ab7774307ce071b294ac1063754fe58433bd85d2" exitCode=0 Nov 02 23:37:41.637182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:41.636801 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" event={"ID":"8d622289-ec5c-4d0a-a380-f5d077ed6b17","Type":"ContainerDied","Data":"b85dbe0a26ea3442ad375622ab7774307ce071b294ac1063754fe58433bd85d2"} Nov 02 23:37:42.789486 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:42.789452 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:42.896325 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:42.896211 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-openshift-ca\") pod \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " Nov 02 23:37:42.896325 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:42.896265 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-xks-ca\") pod \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " Nov 02 23:37:42.896550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:42.896323 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d622289-ec5c-4d0a-a380-f5d077ed6b17-kube-api-access-jt2js\" (UniqueName: \"kubernetes.io/projected/8d622289-ec5c-4d0a-a380-f5d077ed6b17-kube-api-access-jt2js\") pod \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\" (UID: \"8d622289-ec5c-4d0a-a380-f5d077ed6b17\") " Nov 02 23:37:42.897581 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:42.897544 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-openshift-ca" pod "8d622289-ec5c-4d0a-a380-f5d077ed6b17" (UID: "8d622289-ec5c-4d0a-a380-f5d077ed6b17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:42.899076 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:42.899044 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d622289-ec5c-4d0a-a380-f5d077ed6b17-kube-api-access-jt2js" pod "8d622289-ec5c-4d0a-a380-f5d077ed6b17" (UID: "8d622289-ec5c-4d0a-a380-f5d077ed6b17"). InnerVolumeSpecName "kube-api-access-jt2js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:42.899305 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:42.899281 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-xks-ca" pod "8d622289-ec5c-4d0a-a380-f5d077ed6b17" (UID: "8d622289-ec5c-4d0a-a380-f5d077ed6b17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:37:42.998225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:42.998191 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:42.998225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:42.998220 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt2js\" (UniqueName: \"kubernetes.io/projected/8d622289-ec5c-4d0a-a380-f5d077ed6b17-kube-api-access-jt2js\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:42.998225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:42.998230 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d622289-ec5c-4d0a-a380-f5d077ed6b17-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:43.650989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:43.650943 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" event={"ID":"8d622289-ec5c-4d0a-a380-f5d077ed6b17","Type":"ContainerDied","Data":"ad4f8e80aa7cb6663aec54b08e76e81825ba41a66eae72f17b4c4a8a44450d54"} Nov 02 23:37:43.650989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:43.650970 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz" Nov 02 23:37:43.651246 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:43.651004 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad4f8e80aa7cb6663aec54b08e76e81825ba41a66eae72f17b4c4a8a44450d54" Nov 02 23:37:44.597508 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.597469 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl"] Nov 02 23:37:44.598021 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.597960 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8d622289-ec5c-4d0a-a380-f5d077ed6b17" containerName="revoke-keys" Nov 02 23:37:44.598021 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.597980 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d622289-ec5c-4d0a-a380-f5d077ed6b17" containerName="revoke-keys" Nov 02 23:37:44.598177 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.598124 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d622289-ec5c-4d0a-a380-f5d077ed6b17" containerName="revoke-keys" Nov 02 23:37:44.601719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.601694 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:44.658175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.658108 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl"] Nov 02 23:37:44.698605 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.698573 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:37:44.699282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.699243 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:37:44.699345 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.699317 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:37:44.699387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.699378 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:37:44.715354 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.715323 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9h4\" (UniqueName: \"kubernetes.io/projected/20f0ef24-c7ae-4d9e-977d-03de174928d3-kube-api-access-dw9h4\") pod \"maas-api-revoke-sub-1c8781bb-nkjtl\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:44.715514 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.715438 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c8781bb-nkjtl\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:44.715638 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.715609 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c8781bb-nkjtl\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:44.816900 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.816862 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9h4\" (UniqueName: \"kubernetes.io/projected/20f0ef24-c7ae-4d9e-977d-03de174928d3-kube-api-access-dw9h4\") pod \"maas-api-revoke-sub-1c8781bb-nkjtl\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:44.817055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.817002 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c8781bb-nkjtl\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:44.817141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.817068 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c8781bb-nkjtl\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:44.817887 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.817859 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c8781bb-nkjtl\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:44.819555 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.819534 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c8781bb-nkjtl\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:44.870077 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.870016 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9h4\" (UniqueName: \"kubernetes.io/projected/20f0ef24-c7ae-4d9e-977d-03de174928d3-kube-api-access-dw9h4\") pod \"maas-api-revoke-sub-1c8781bb-nkjtl\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:44.920280 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:44.920252 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:45.298001 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:45.297967 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl"] Nov 02 23:37:45.299844 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:37:45.299660 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f0ef24_c7ae_4d9e_977d_03de174928d3.slice/crio-0f32251a7bef1ffd45530f214c7f323749f34b98c9fac1c3b7ae9deb8a7f0093 WatchSource:0}: Error finding container 0f32251a7bef1ffd45530f214c7f323749f34b98c9fac1c3b7ae9deb8a7f0093: Status 404 returned error can't find the container with id 0f32251a7bef1ffd45530f214c7f323749f34b98c9fac1c3b7ae9deb8a7f0093 Nov 02 23:37:45.668179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:45.668128 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" event={"ID":"20f0ef24-c7ae-4d9e-977d-03de174928d3","Type":"ContainerStarted","Data":"0f32251a7bef1ffd45530f214c7f323749f34b98c9fac1c3b7ae9deb8a7f0093"} Nov 02 23:37:45.817664 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:45.817592 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77"] Nov 02 23:37:45.821554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:45.821532 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:45.877164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:45.877049 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77"] Nov 02 23:37:45.931685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:45.931641 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33dea87-rcd77\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:45.931882 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:45.931730 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33dea87-rcd77\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:45.931882 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:45.931766 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9crnw\" (UniqueName: \"kubernetes.io/projected/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-kube-api-access-9crnw\") pod \"maas-api-revoke-sub-d33dea87-rcd77\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:46.032683 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:46.032650 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33dea87-rcd77\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:46.032856 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:46.032698 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9crnw\" (UniqueName: \"kubernetes.io/projected/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-kube-api-access-9crnw\") pod \"maas-api-revoke-sub-d33dea87-rcd77\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:46.032856 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:46.032793 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33dea87-rcd77\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:46.033429 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:46.033404 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33dea87-rcd77\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:46.034976 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:46.034954 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33dea87-rcd77\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:46.085685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:46.085616 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9crnw\" (UniqueName: \"kubernetes.io/projected/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-kube-api-access-9crnw\") pod \"maas-api-revoke-sub-d33dea87-rcd77\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:46.141649 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:46.141619 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:46.542193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:46.542161 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77"] Nov 02 23:37:46.543089 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:37:46.543050 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf263724_6e40_4a97_abc5_3c8cdff3ff2f.slice/crio-7b0f9b942270a61d1c39beba7ec7bc116c3d3db20e6d605d27383dcd3ed8117f WatchSource:0}: Error finding container 7b0f9b942270a61d1c39beba7ec7bc116c3d3db20e6d605d27383dcd3ed8117f: Status 404 returned error can't find the container with id 7b0f9b942270a61d1c39beba7ec7bc116c3d3db20e6d605d27383dcd3ed8117f Nov 02 23:37:46.676472 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:46.676436 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" event={"ID":"cf263724-6e40-4a97-abc5-3c8cdff3ff2f","Type":"ContainerStarted","Data":"7b0f9b942270a61d1c39beba7ec7bc116c3d3db20e6d605d27383dcd3ed8117f"} Nov 02 23:37:46.678040 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:46.678014 5672 generic.go:356] "Generic (PLEG): container finished" podID="20f0ef24-c7ae-4d9e-977d-03de174928d3" containerID="3f9f8f5957488eff74c6ab5ca76a9a023b3b2ae612f96babdb55cdc7ded5f453" exitCode=0 Nov 02 23:37:46.678211 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:46.678104 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" event={"ID":"20f0ef24-c7ae-4d9e-977d-03de174928d3","Type":"ContainerDied","Data":"3f9f8f5957488eff74c6ab5ca76a9a023b3b2ae612f96babdb55cdc7ded5f453"} Nov 02 23:37:47.686262 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:47.686168 5672 generic.go:356] "Generic (PLEG): container finished" podID="cf263724-6e40-4a97-abc5-3c8cdff3ff2f" containerID="c184dba65f550b42de36864c9a97b15c4d27b42c7bbea4bb4b25432a2470442b" exitCode=0 Nov 02 23:37:47.686739 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:47.686263 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" event={"ID":"cf263724-6e40-4a97-abc5-3c8cdff3ff2f","Type":"ContainerDied","Data":"c184dba65f550b42de36864c9a97b15c4d27b42c7bbea4bb4b25432a2470442b"} Nov 02 23:37:47.867875 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:47.867844 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:47.959334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:47.959253 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-openshift-ca\") pod \"20f0ef24-c7ae-4d9e-977d-03de174928d3\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " Nov 02 23:37:47.959334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:47.959319 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f0ef24-c7ae-4d9e-977d-03de174928d3-kube-api-access-dw9h4\" (UniqueName: \"kubernetes.io/projected/20f0ef24-c7ae-4d9e-977d-03de174928d3-kube-api-access-dw9h4\") pod \"20f0ef24-c7ae-4d9e-977d-03de174928d3\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " Nov 02 23:37:47.959563 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:47.959482 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-xks-ca\") pod \"20f0ef24-c7ae-4d9e-977d-03de174928d3\" (UID: \"20f0ef24-c7ae-4d9e-977d-03de174928d3\") " Nov 02 23:37:47.959672 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:47.959651 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-openshift-ca" pod "20f0ef24-c7ae-4d9e-977d-03de174928d3" (UID: "20f0ef24-c7ae-4d9e-977d-03de174928d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:47.959959 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:47.959937 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:47.961491 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:47.961457 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f0ef24-c7ae-4d9e-977d-03de174928d3-kube-api-access-dw9h4" pod "20f0ef24-c7ae-4d9e-977d-03de174928d3" (UID: "20f0ef24-c7ae-4d9e-977d-03de174928d3"). InnerVolumeSpecName "kube-api-access-dw9h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:47.961611 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:47.961545 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-xks-ca" pod "20f0ef24-c7ae-4d9e-977d-03de174928d3" (UID: "20f0ef24-c7ae-4d9e-977d-03de174928d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:37:48.044931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.044898 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns"] Nov 02 23:37:48.047885 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.047853 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bd3cd6-8v2ns"] Nov 02 23:37:48.060826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.060795 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9h4\" (UniqueName: \"kubernetes.io/projected/20f0ef24-c7ae-4d9e-977d-03de174928d3-kube-api-access-dw9h4\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:48.060929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.060832 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f0ef24-c7ae-4d9e-977d-03de174928d3-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:48.292978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.292942 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl"] Nov 02 23:37:48.293627 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.293610 5672 cpu_manager.go:414] "containerMap: removing container" podUID="20f0ef24-c7ae-4d9e-977d-03de174928d3" containerName="revoke-keys" Nov 02 23:37:48.293681 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.293628 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f0ef24-c7ae-4d9e-977d-03de174928d3" containerName="revoke-keys" Nov 02 23:37:48.293794 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.293748 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f0ef24-c7ae-4d9e-977d-03de174928d3" containerName="revoke-keys" Nov 02 23:37:48.297417 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.297397 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 02 23:37:48.347542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.347389 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c5148c-dockercfg-tbjxp\"" Nov 02 23:37:48.347877 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.347848 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c5148c\"" Nov 02 23:37:48.349463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.349440 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl"] Nov 02 23:37:48.465590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.465557 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djk9c\" (UniqueName: \"kubernetes.io/projected/3941edda-d840-46e5-b257-d42202cd8214-kube-api-access-djk9c\") pod \"payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl\" (UID: \"3941edda-d840-46e5-b257-d42202cd8214\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 02 23:37:48.465764 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.465739 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3941edda-d840-46e5-b257-d42202cd8214-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl\" (UID: \"3941edda-d840-46e5-b257-d42202cd8214\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 02 23:37:48.569680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.567786 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3941edda-d840-46e5-b257-d42202cd8214-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl\" (UID: \"3941edda-d840-46e5-b257-d42202cd8214\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 02 23:37:48.569680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.568125 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djk9c\" (UniqueName: \"kubernetes.io/projected/3941edda-d840-46e5-b257-d42202cd8214-kube-api-access-djk9c\") pod \"payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl\" (UID: \"3941edda-d840-46e5-b257-d42202cd8214\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 02 23:37:48.569680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.569051 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3941edda-d840-46e5-b257-d42202cd8214-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl\" (UID: \"3941edda-d840-46e5-b257-d42202cd8214\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 02 23:37:48.624095 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.624064 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djk9c\" (UniqueName: \"kubernetes.io/projected/3941edda-d840-46e5-b257-d42202cd8214-kube-api-access-djk9c\") pod \"payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl\" (UID: \"3941edda-d840-46e5-b257-d42202cd8214\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 02 23:37:48.695380 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.695333 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" event={"ID":"20f0ef24-c7ae-4d9e-977d-03de174928d3","Type":"ContainerDied","Data":"0f32251a7bef1ffd45530f214c7f323749f34b98c9fac1c3b7ae9deb8a7f0093"} Nov 02 23:37:48.695805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.695426 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f32251a7bef1ffd45530f214c7f323749f34b98c9fac1c3b7ae9deb8a7f0093" Nov 02 23:37:48.695805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.695450 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl" Nov 02 23:37:48.730198 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.726867 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198bb295-f72a-421b-951d-17e09725d8aa" path="/var/lib/kubelet/pods/198bb295-f72a-421b-951d-17e09725d8aa/volumes" Nov 02 23:37:48.873613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.873586 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:48.912680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.912649 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 02 23:37:48.976835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.973921 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-xks-ca\") pod \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " Nov 02 23:37:48.976835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.974076 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-openshift-ca\") pod \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " Nov 02 23:37:48.976835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.974210 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-kube-api-access-9crnw\" (UniqueName: \"kubernetes.io/projected/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-kube-api-access-9crnw\") pod \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\" (UID: \"cf263724-6e40-4a97-abc5-3c8cdff3ff2f\") " Nov 02 23:37:48.977512 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.977331 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-openshift-ca" pod "cf263724-6e40-4a97-abc5-3c8cdff3ff2f" (UID: "cf263724-6e40-4a97-abc5-3c8cdff3ff2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:37:48.977662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.977584 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-kube-api-access-9crnw" pod "cf263724-6e40-4a97-abc5-3c8cdff3ff2f" (UID: "cf263724-6e40-4a97-abc5-3c8cdff3ff2f"). InnerVolumeSpecName "kube-api-access-9crnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:37:48.999014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:48.998984 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-xks-ca" pod "cf263724-6e40-4a97-abc5-3c8cdff3ff2f" (UID: "cf263724-6e40-4a97-abc5-3c8cdff3ff2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:37:49.075379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:49.075334 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:49.075379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:49.075376 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9crnw\" (UniqueName: \"kubernetes.io/projected/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-kube-api-access-9crnw\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:49.075379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:49.075389 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf263724-6e40-4a97-abc5-3c8cdff3ff2f-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:37:49.290119 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:49.290088 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl"] Nov 02 23:37:49.290605 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:37:49.290575 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3941edda_d840_46e5_b257_d42202cd8214.slice/crio-7fb5f23139365ac9ab2d4dd0d9a6a5f47ce060ac37e2b3d26a731e3f1dc5bf0f WatchSource:0}: Error finding container 7fb5f23139365ac9ab2d4dd0d9a6a5f47ce060ac37e2b3d26a731e3f1dc5bf0f: Status 404 returned error can't find the container with id 7fb5f23139365ac9ab2d4dd0d9a6a5f47ce060ac37e2b3d26a731e3f1dc5bf0f Nov 02 23:37:49.704171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:49.704066 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" event={"ID":"3941edda-d840-46e5-b257-d42202cd8214","Type":"ContainerStarted","Data":"e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee"} Nov 02 23:37:49.704171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:49.704135 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 02 23:37:49.704633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:49.704176 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" event={"ID":"3941edda-d840-46e5-b257-d42202cd8214","Type":"ContainerStarted","Data":"7fb5f23139365ac9ab2d4dd0d9a6a5f47ce060ac37e2b3d26a731e3f1dc5bf0f"} Nov 02 23:37:49.705564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:49.705539 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" event={"ID":"cf263724-6e40-4a97-abc5-3c8cdff3ff2f","Type":"ContainerDied","Data":"7b0f9b942270a61d1c39beba7ec7bc116c3d3db20e6d605d27383dcd3ed8117f"} Nov 02 23:37:49.705675 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:49.705578 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b0f9b942270a61d1c39beba7ec7bc116c3d3db20e6d605d27383dcd3ed8117f" Nov 02 23:37:49.705675 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:49.705553 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77" Nov 02 23:37:49.810936 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:49.810879 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" podStartSLOduration=1.810863815 podStartE2EDuration="1.810863815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:49.809783524 +0000 UTC m=+2663.217095354" watchObservedRunningTime="2026-11-02 23:37:49.810863815 +0000 UTC m=+2663.218175692" Nov 02 23:37:56.021451 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:37:56.021415 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:37:58.997868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:37:58.997831 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:38:00.720547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:00.720520 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 02 23:38:01.374704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.374664 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx"] Nov 02 23:38:01.375144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.375127 5672 cpu_manager.go:414] "containerMap: removing container" podUID="cf263724-6e40-4a97-abc5-3c8cdff3ff2f" containerName="revoke-keys" Nov 02 23:38:01.375144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.375156 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf263724-6e40-4a97-abc5-3c8cdff3ff2f" containerName="revoke-keys" Nov 02 23:38:01.375287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.375266 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf263724-6e40-4a97-abc5-3c8cdff3ff2f" containerName="revoke-keys" Nov 02 23:38:01.380134 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.380114 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 02 23:38:01.424044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.424005 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8k9\" (UniqueName: \"kubernetes.io/projected/5af13ee4-e8a6-4332-9601-4b9f243625b3-kube-api-access-tz8k9\") pod \"payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx\" (UID: \"5af13ee4-e8a6-4332-9601-4b9f243625b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 02 23:38:01.424356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.424321 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af13ee4-e8a6-4332-9601-4b9f243625b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx\" (UID: \"5af13ee4-e8a6-4332-9601-4b9f243625b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 02 23:38:01.438073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.438041 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls"] Nov 02 23:38:01.449906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.449875 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx"] Nov 02 23:38:01.450086 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.449997 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.474705 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.474676 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-fe63ac\"" Nov 02 23:38:01.475213 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.475188 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-fe63ac-dockercfg-88dps\"" Nov 02 23:38:01.494113 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.494080 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls"] Nov 02 23:38:01.525795 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.525751 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-envoy\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.526094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.526067 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af13ee4-e8a6-4332-9601-4b9f243625b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx\" (UID: \"5af13ee4-e8a6-4332-9601-4b9f243625b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 02 23:38:01.526287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.526138 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq87s\" (UniqueName: \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-kube-api-access-lq87s\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.526287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.526224 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-podinfo\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.526287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.526257 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-token\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.526287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.526285 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-socket\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.526537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.526358 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-data\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.526537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.526402 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-credential-socket\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.526537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.526438 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-certs\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.526537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.526477 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8k9\" (UniqueName: \"kubernetes.io/projected/5af13ee4-e8a6-4332-9601-4b9f243625b3-kube-api-access-tz8k9\") pod \"payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx\" (UID: \"5af13ee4-e8a6-4332-9601-4b9f243625b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 02 23:38:01.526537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.526504 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istiod-ca-cert\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.527482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.527453 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af13ee4-e8a6-4332-9601-4b9f243625b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx\" (UID: \"5af13ee4-e8a6-4332-9601-4b9f243625b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 02 23:38:01.533430 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.533193 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-28f82c-openshift-default-dockercfg-2qz6x\"" Nov 02 23:38:01.581490 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.581461 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8k9\" (UniqueName: \"kubernetes.io/projected/5af13ee4-e8a6-4332-9601-4b9f243625b3-kube-api-access-tz8k9\") pod \"payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx\" (UID: \"5af13ee4-e8a6-4332-9601-4b9f243625b3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.644644 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq87s\" (UniqueName: \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-kube-api-access-lq87s\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.644722 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-podinfo\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.644757 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-token\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.644782 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-socket\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.644849 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-data\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.644893 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-credential-socket\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.644927 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-certs\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.644965 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istiod-ca-cert\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.645048 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-envoy\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.645850 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-socket\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.645997 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-credential-socket\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.646312 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-data\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.650463 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-certs\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.654219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.650739 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istiod-ca-cert\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.675403 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.669558 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-envoy\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.675403 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.669745 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-podinfo\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.695108 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.694124 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 02 23:38:01.700996 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.700636 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq87s\" (UniqueName: \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-kube-api-access-lq87s\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.700996 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.700944 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-token\") pod \"e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:01.775557 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:01.775376 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:02.100387 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:38:02.100347 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af13ee4_e8a6_4332_9601_4b9f243625b3.slice/crio-5092b63635d621951353e9cce19de18403ae269045ca176085168a1d5b9df0d2 WatchSource:0}: Error finding container 5092b63635d621951353e9cce19de18403ae269045ca176085168a1d5b9df0d2: Status 404 returned error can't find the container with id 5092b63635d621951353e9cce19de18403ae269045ca176085168a1d5b9df0d2 Nov 02 23:38:02.101888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:02.101843 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx"] Nov 02 23:38:02.215482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:02.215449 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls"] Nov 02 23:38:02.215864 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:38:02.215839 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adcfc0e_353b_44e2_b6ee_8a915fac9b52.slice/crio-0aa6fcf77e69c57206220f5b8d5095193b1693944a514f0c2bbd765cbbb1832b WatchSource:0}: Error finding container 0aa6fcf77e69c57206220f5b8d5095193b1693944a514f0c2bbd765cbbb1832b: Status 404 returned error can't find the container with id 0aa6fcf77e69c57206220f5b8d5095193b1693944a514f0c2bbd765cbbb1832b Nov 02 23:38:02.218546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:02.218507 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:02.218635 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:02.218592 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:02.218674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:02.218637 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:02.821222 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:02.821176 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" event={"ID":"5af13ee4-e8a6-4332-9601-4b9f243625b3","Type":"ContainerStarted","Data":"1164a8344a87e5579761d78ed1f3b44cdfb513e7f41bca1481598e7a8be68719"} Nov 02 23:38:02.821744 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:02.821239 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" event={"ID":"5af13ee4-e8a6-4332-9601-4b9f243625b3","Type":"ContainerStarted","Data":"5092b63635d621951353e9cce19de18403ae269045ca176085168a1d5b9df0d2"} Nov 02 23:38:02.821744 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:02.821292 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 02 23:38:02.823107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:02.823076 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" event={"ID":"9adcfc0e-353b-44e2-b6ee-8a915fac9b52","Type":"ContainerStarted","Data":"546303a23c4d2730478e14f596e7ca9561ee2e58eb63df3e605736acb1e2aa24"} Nov 02 23:38:02.823227 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:02.823120 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" event={"ID":"9adcfc0e-353b-44e2-b6ee-8a915fac9b52","Type":"ContainerStarted","Data":"0aa6fcf77e69c57206220f5b8d5095193b1693944a514f0c2bbd765cbbb1832b"} Nov 02 23:38:02.933723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:02.932895 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" podStartSLOduration=1.932877146 podStartE2EDuration="1.932877146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:02.927583405 +0000 UTC m=+2676.334895236" watchObservedRunningTime="2026-11-02 23:38:02.932877146 +0000 UTC m=+2676.340188976" Nov 02 23:38:03.041338 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:03.041260 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" podStartSLOduration=2.041242151 podStartE2EDuration="2.041242151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:03.036658326 +0000 UTC m=+2676.443970156" watchObservedRunningTime="2026-11-02 23:38:03.041242151 +0000 UTC m=+2676.448553982" Nov 02 23:38:03.776667 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:03.776631 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:03.782196 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:03.782170 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:03.832050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:03.832015 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:03.833283 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:03.833257 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:38:06.045826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:06.045779 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz"] Nov 02 23:38:06.090587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:06.090551 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1387e85c-b6tjz"] Nov 02 23:38:06.712486 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:06.712456 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279470ef-32ae-4787-8580-2e1d095204f0" path="/var/lib/kubelet/pods/279470ef-32ae-4787-8580-2e1d095204f0/volumes" Nov 02 23:38:13.834587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:13.834560 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 02 23:38:21.618760 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.618711 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk"] Nov 02 23:38:21.624594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.624570 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.664843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.664805 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9"] Nov 02 23:38:21.669349 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.669324 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.673899 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.673877 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fb6e1d-openshift-default-dockercfg-5lz5h\"" Nov 02 23:38:21.721142 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.721107 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk"] Nov 02 23:38:21.782204 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.782169 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9"] Nov 02 23:38:21.788766 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.788736 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996905d6-94af-41c4-915c-e69dfc9688e2-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.789039 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.789017 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvp2\" (UniqueName: \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-kube-api-access-plvp2\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.789229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.789211 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8604f1-7218-42b8-9dff-499f12b878c2-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.789409 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.789394 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.789570 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.789555 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.789742 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.789727 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.789894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.789873 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.790050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.790035 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.790230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.790214 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.790416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.790401 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.790550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.790537 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2spz\" (UniqueName: \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-kube-api-access-p2spz\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.790719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.790697 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.790906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.790890 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.791075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.791052 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996905d6-94af-41c4-915c-e69dfc9688e2-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.791466 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.791444 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.791697 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.791681 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.791948 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.791925 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.792778 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.792760 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.894459 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.894368 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.894459 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.894418 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.894754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.894543 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2spz\" (UniqueName: \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-kube-api-access-p2spz\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.894754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.894608 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.894754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.894652 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.894754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.894704 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996905d6-94af-41c4-915c-e69dfc9688e2-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.894754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.894753 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.895060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.894779 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.895060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.894802 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.895060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.894810 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.895060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.894988 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.895060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895038 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.895382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895042 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.895382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895112 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.895382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895168 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996905d6-94af-41c4-915c-e69dfc9688e2-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.895382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895207 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvp2\" (UniqueName: \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-kube-api-access-plvp2\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.895382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895236 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8604f1-7218-42b8-9dff-499f12b878c2-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.895382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895258 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.895382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895143 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.895382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895298 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.895382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895337 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.895881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895405 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.895881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895417 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996905d6-94af-41c4-915c-e69dfc9688e2-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.895881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895439 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.895881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895488 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.895881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895555 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.895881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895799 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.896246 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.895932 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8604f1-7218-42b8-9dff-499f12b878c2-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.897474 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.897445 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.897640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.897622 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.898259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.898238 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.898370 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.898240 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996905d6-94af-41c4-915c-e69dfc9688e2-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.950735 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.950708 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvp2\" (UniqueName: \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-kube-api-access-plvp2\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.950955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.950933 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:21.951082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.951061 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2spz\" (UniqueName: \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-kube-api-access-p2spz\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.951407 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.951386 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:21.987378 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:21.987356 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:22.055590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.055379 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x"] Nov 02 23:38:22.059494 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.059466 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1a657c-n9k6x"] Nov 02 23:38:22.240921 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.240890 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:22.403503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.403473 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9"] Nov 02 23:38:22.403947 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:38:22.403914 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996905d6_94af_41c4_915c_e69dfc9688e2.slice/crio-da59dd26874f8b287472fa7be553c0c7860b4c92858415b4b4eec7842558ee4b WatchSource:0}: Error finding container da59dd26874f8b287472fa7be553c0c7860b4c92858415b4b4eec7842558ee4b: Status 404 returned error can't find the container with id da59dd26874f8b287472fa7be553c0c7860b4c92858415b4b4eec7842558ee4b Nov 02 23:38:22.406644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.406613 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:22.406726 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.406676 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:22.406726 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.406710 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:22.618081 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.618050 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk"] Nov 02 23:38:22.625440 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.625387 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:22.627122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.625537 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:22.627122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.625627 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:38:22.714274 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.714243 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd134bca-8f4a-4b50-93f6-b6f7b9565877" path="/var/lib/kubelet/pods/cd134bca-8f4a-4b50-93f6-b6f7b9565877/volumes" Nov 02 23:38:22.983044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.982998 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" event={"ID":"996905d6-94af-41c4-915c-e69dfc9688e2","Type":"ContainerStarted","Data":"5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e"} Nov 02 23:38:22.983269 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.983068 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" event={"ID":"996905d6-94af-41c4-915c-e69dfc9688e2","Type":"ContainerStarted","Data":"da59dd26874f8b287472fa7be553c0c7860b4c92858415b4b4eec7842558ee4b"} Nov 02 23:38:22.985401 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.985368 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" event={"ID":"fc8604f1-7218-42b8-9dff-499f12b878c2","Type":"ContainerStarted","Data":"40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a"} Nov 02 23:38:22.985534 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.985408 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" event={"ID":"fc8604f1-7218-42b8-9dff-499f12b878c2","Type":"ContainerStarted","Data":"c4e4d6a682bba053e43ffe9d6daa849402274b6700a76821209511dfb17cb0d8"} Nov 02 23:38:22.987665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:22.987645 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:23.095387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:23.095320 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" podStartSLOduration=2.09530114 podStartE2EDuration="2.09530114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:23.093326585 +0000 UTC m=+2696.500638415" watchObservedRunningTime="2026-11-02 23:38:23.09530114 +0000 UTC m=+2696.502612972" Nov 02 23:38:23.213862 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:23.213798 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" podStartSLOduration=2.21378347 podStartE2EDuration="2.21378347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:23.211554813 +0000 UTC m=+2696.618866644" watchObservedRunningTime="2026-11-02 23:38:23.21378347 +0000 UTC m=+2696.621095370" Nov 02 23:38:23.245230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:23.245109 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:23.245425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:23.245312 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 23:38:23.245425 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:23.245364 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 23:38:23.993302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:23.992820 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 02 23:38:23.993302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:23.992901 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" podUID="996905d6-94af-41c4-915c-e69dfc9688e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 02 23:38:24.245273 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:24.244549 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 23:38:24.245273 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:24.244614 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 23:38:24.988632 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:24.988584 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 02 23:38:24.988849 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:24.988645 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" podUID="996905d6-94af-41c4-915c-e69dfc9688e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 02 23:38:25.242225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:25.241564 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 23:38:25.242225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:25.241636 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 23:38:25.989806 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:25.989769 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 02 23:38:25.990612 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:25.990104 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" podUID="996905d6-94af-41c4-915c-e69dfc9688e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 02 23:38:26.015853 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.015828 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv"] Nov 02 23:38:26.020202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.020180 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:38:26.058172 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:38:26.054777 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:38:26.087203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.086887 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv"] Nov 02 23:38:26.108291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.108262 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e86d19-a8c6-44b0-9076-f47f4221b525-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv\" (UID: \"68e86d19-a8c6-44b0-9076-f47f4221b525\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:38:26.108438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.108317 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt7mq\" (UniqueName: \"kubernetes.io/projected/68e86d19-a8c6-44b0-9076-f47f4221b525-kube-api-access-tt7mq\") pod \"payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv\" (UID: \"68e86d19-a8c6-44b0-9076-f47f4221b525\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:38:26.112492 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.112467 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b1be95-dockercfg-qf9v9\"" Nov 02 23:38:26.112618 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.112564 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b1be95\"" Nov 02 23:38:26.209932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.209892 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e86d19-a8c6-44b0-9076-f47f4221b525-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv\" (UID: \"68e86d19-a8c6-44b0-9076-f47f4221b525\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:38:26.211100 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.211078 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt7mq\" (UniqueName: \"kubernetes.io/projected/68e86d19-a8c6-44b0-9076-f47f4221b525-kube-api-access-tt7mq\") pod \"payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv\" (UID: \"68e86d19-a8c6-44b0-9076-f47f4221b525\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:38:26.211731 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.210990 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e86d19-a8c6-44b0-9076-f47f4221b525-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv\" (UID: \"68e86d19-a8c6-44b0-9076-f47f4221b525\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:38:26.242649 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.242555 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 23:38:26.243252 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.243221 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 23:38:26.277007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.276975 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt7mq\" (UniqueName: \"kubernetes.io/projected/68e86d19-a8c6-44b0-9076-f47f4221b525-kube-api-access-tt7mq\") pod \"payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv\" (UID: \"68e86d19-a8c6-44b0-9076-f47f4221b525\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:38:26.390447 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.390407 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:38:26.898269 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.898237 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv"] Nov 02 23:38:26.988581 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.988551 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 02 23:38:26.988716 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:26.988605 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" podUID="996905d6-94af-41c4-915c-e69dfc9688e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 02 23:38:27.020872 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:27.020843 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" event={"ID":"68e86d19-a8c6-44b0-9076-f47f4221b525","Type":"ContainerStarted","Data":"f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92"} Nov 02 23:38:27.020983 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:27.020900 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" event={"ID":"68e86d19-a8c6-44b0-9076-f47f4221b525","Type":"ContainerStarted","Data":"ca33d360be4b7427cac2af92028d1ad60ff02b3017c555f21dcc9be4619ed26c"} Nov 02 23:38:27.020983 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:27.020938 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:38:27.161221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:27.159607 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" podStartSLOduration=2.159586113 podStartE2EDuration="2.159586113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:27.143436061 +0000 UTC m=+2700.550747891" watchObservedRunningTime="2026-11-02 23:38:27.159586113 +0000 UTC m=+2700.566897944" Nov 02 23:38:27.262172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:27.251708 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 23:38:27.262172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:27.251767 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 23:38:27.989043 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:27.988570 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 02 23:38:27.989043 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:27.988628 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" podUID="996905d6-94af-41c4-915c-e69dfc9688e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 02 23:38:28.241613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:28.241520 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 23:38:28.241877 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:28.241844 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 23:38:28.988083 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:28.988051 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 02 23:38:28.988583 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:28.988101 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" podUID="996905d6-94af-41c4-915c-e69dfc9688e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 02 23:38:29.242371 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:29.242278 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 23:38:29.242371 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:29.242357 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 23:38:29.993820 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:29.993789 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:29.994340 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:29.994320 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:29.995178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:29.995140 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:38:30.246300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:30.246229 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:30.246503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:30.246483 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:30.247310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:30.247291 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:38:36.822260 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:36.822183 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:38:38.046235 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:38.046208 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:38:39.045078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:39.045027 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8"] Nov 02 23:38:39.048323 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:39.048292 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6fb8e42-hw6h8"] Nov 02 23:38:40.713092 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:40.713058 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fbdc94-3d1e-460a-8a39-188bc40b81a9" path="/var/lib/kubelet/pods/26fbdc94-3d1e-460a-8a39-188bc40b81a9/volumes" Nov 02 23:38:49.052143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:49.052103 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm"] Nov 02 23:38:49.097972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:49.097935 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3256c5-dw9nm"] Nov 02 23:38:50.713358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:38:50.713320 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc" path="/var/lib/kubelet/pods/1340a2c0-2f2f-4836-a8bd-2f3d3cac4efc/volumes" Nov 02 23:38:56.021190 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:38:56.021135 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:39:01.045499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:01.045458 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq"] Nov 02 23:39:01.091254 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:01.091224 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c207868-clljq"] Nov 02 23:39:02.653687 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:02.653646 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:39:02.713857 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:02.713823 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d66e697-a76a-4847-8635-ef632da18e0b" path="/var/lib/kubelet/pods/2d66e697-a76a-4847-8635-ef632da18e0b/volumes" Nov 02 23:39:07.046220 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:07.046177 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9"] Nov 02 23:39:07.091954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:07.091921 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e9c50cd-gbhc9"] Nov 02 23:39:08.712109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:08.712078 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aed28f6-cf92-44e9-b6bd-5b3297e5208c" path="/var/lib/kubelet/pods/0aed28f6-cf92-44e9-b6bd-5b3297e5208c/volumes" Nov 02 23:39:18.045598 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:18.045564 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr"] Nov 02 23:39:18.093162 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:18.093124 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6777eb6-z56fr"] Nov 02 23:39:18.720076 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:18.720033 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938f8589-bbe7-4ef6-8fe8-43c81f550222" path="/var/lib/kubelet/pods/938f8589-bbe7-4ef6-8fe8-43c81f550222/volumes" Nov 02 23:39:26.021108 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:39:26.021064 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:39:28.388452 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:28.388414 5672 scope.go:122] "RemoveContainer" containerID="4f06956f87d483fe25c36adf5b177d38b47aef87398cb4b1c66ac9534a5e2e3f" Nov 02 23:39:28.398571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:28.398546 5672 scope.go:122] "RemoveContainer" containerID="ab15b9aa251555289d799d5111a1871c4967fafa1dedc9b1c549ea70ef577a2a" Nov 02 23:39:28.408201 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:28.408177 5672 scope.go:122] "RemoveContainer" containerID="fbcdc69b27ae2e6481873fb818da549acb50ba35ea3a5fc4b4087ff41557ab57" Nov 02 23:39:29.669555 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:29.669525 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:39:32.445526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.444036 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh"] Nov 02 23:39:32.455260 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.453086 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.465383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.465353 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-credential-socket\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.465590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.465571 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-socket\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.465717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.465700 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq89f\" (UniqueName: \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-kube-api-access-sq89f\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.465874 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.465857 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed3e348-d810-43a0-afa2-cf8083bc8874-istiod-ca-cert\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.466478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.466458 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-podinfo\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.466644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.466624 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-certs\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.466777 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.466760 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-data\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.466939 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.466923 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-token\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.467073 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.467044 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-envoy\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.561709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.561676 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fbec82-openshift-default-dockercfg-nf6wj\"" Nov 02 23:39:32.688522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.622648 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-token\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.688731 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.688577 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-envoy\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.688731 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.688720 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-credential-socket\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.688859 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.688772 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-socket\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.688859 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.688807 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq89f\" (UniqueName: \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-kube-api-access-sq89f\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.688980 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.688896 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed3e348-d810-43a0-afa2-cf8083bc8874-istiod-ca-cert\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.688980 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.688950 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-podinfo\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.689087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.688991 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-certs\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.689087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.689026 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-data\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.689309 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.689283 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-credential-socket\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.689501 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.689475 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-data\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.689583 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.689500 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-socket\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.689764 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.689744 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-certs\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.690234 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.690211 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed3e348-d810-43a0-afa2-cf8083bc8874-istiod-ca-cert\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.690625 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.690605 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh"] Nov 02 23:39:32.695878 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.695791 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-podinfo\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.700117 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.698108 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-envoy\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.738009 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.737977 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-token\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.756287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.756255 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq89f\" (UniqueName: \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-kube-api-access-sq89f\") pod \"e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:32.798797 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:32.798754 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:33.067706 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:33.063603 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9"] Nov 02 23:39:33.069242 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:33.069214 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b0fc8e-q6rn9"] Nov 02 23:39:33.265587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:33.263799 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh"] Nov 02 23:39:33.276992 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:33.276945 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:33.277133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:33.277043 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:33.277133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:33.277091 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:33.595013 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:33.594918 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" event={"ID":"7ed3e348-d810-43a0-afa2-cf8083bc8874","Type":"ContainerStarted","Data":"ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b"} Nov 02 23:39:33.595013 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:33.594977 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" event={"ID":"7ed3e348-d810-43a0-afa2-cf8083bc8874","Type":"ContainerStarted","Data":"d42154935c322940910e879f16a74cc2278761a2754333c1eed3fa7cff28a346"} Nov 02 23:39:33.706863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:33.706799 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" podStartSLOduration=1.70678038 podStartE2EDuration="1.70678038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:39:33.705881898 +0000 UTC m=+2767.113193727" watchObservedRunningTime="2026-11-02 23:39:33.70678038 +0000 UTC m=+2767.114092211" Nov 02 23:39:33.800413 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:33.800375 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:34.712882 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:34.712847 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90486d35-1f87-46b4-b4ee-23f26ceb47d3" path="/var/lib/kubelet/pods/90486d35-1f87-46b4-b4ee-23f26ceb47d3/volumes" Nov 02 23:39:34.804620 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:34.804588 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:35.608542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:35.608513 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:35.609351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:35.609332 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:39:40.117367 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.117284 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7"] Nov 02 23:39:40.121331 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.121310 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 02 23:39:40.174792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.174761 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7"] Nov 02 23:39:40.197043 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.197012 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr9vt\" (UniqueName: \"kubernetes.io/projected/11791814-60d7-405a-ae75-ff0b9e96983a-kube-api-access-vr9vt\") pod \"payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7\" (UID: \"11791814-60d7-405a-ae75-ff0b9e96983a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 02 23:39:40.197256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.197233 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11791814-60d7-405a-ae75-ff0b9e96983a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7\" (UID: \"11791814-60d7-405a-ae75-ff0b9e96983a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 02 23:39:40.216954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.216864 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-fbec82\"" Nov 02 23:39:40.216954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.216893 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-fbec82-dockercfg-4z8b7\"" Nov 02 23:39:40.290522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.290482 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf"] Nov 02 23:39:40.295327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.295282 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 02 23:39:40.301104 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.300940 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11791814-60d7-405a-ae75-ff0b9e96983a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7\" (UID: \"11791814-60d7-405a-ae75-ff0b9e96983a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 02 23:39:40.301531 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.301505 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr9vt\" (UniqueName: \"kubernetes.io/projected/11791814-60d7-405a-ae75-ff0b9e96983a-kube-api-access-vr9vt\") pod \"payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7\" (UID: \"11791814-60d7-405a-ae75-ff0b9e96983a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 02 23:39:40.302837 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.302789 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11791814-60d7-405a-ae75-ff0b9e96983a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7\" (UID: \"11791814-60d7-405a-ae75-ff0b9e96983a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 02 23:39:40.340521 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.340486 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf"] Nov 02 23:39:40.357822 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.357797 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr9vt\" (UniqueName: \"kubernetes.io/projected/11791814-60d7-405a-ae75-ff0b9e96983a-kube-api-access-vr9vt\") pod \"payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7\" (UID: \"11791814-60d7-405a-ae75-ff0b9e96983a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 02 23:39:40.402313 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.402283 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtsf\" (UniqueName: \"kubernetes.io/projected/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-kube-api-access-4dtsf\") pod \"payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf\" (UID: \"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 02 23:39:40.402495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.402348 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf\" (UID: \"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 02 23:39:40.438517 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.438495 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 02 23:39:40.504566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.503529 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf\" (UID: \"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 02 23:39:40.504566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.503791 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtsf\" (UniqueName: \"kubernetes.io/projected/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-kube-api-access-4dtsf\") pod \"payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf\" (UID: \"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 02 23:39:40.505060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.504986 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf\" (UID: \"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 02 23:39:40.556887 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.556850 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtsf\" (UniqueName: \"kubernetes.io/projected/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-kube-api-access-4dtsf\") pod \"payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf\" (UID: \"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 02 23:39:40.614958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.614922 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 02 23:39:40.663201 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.663136 5672 generic.go:356] "Generic (PLEG): container finished" podID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerID="546303a23c4d2730478e14f596e7ca9561ee2e58eb63df3e605736acb1e2aa24" exitCode=0 Nov 02 23:39:40.663383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.663197 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" event={"ID":"9adcfc0e-353b-44e2-b6ee-8a915fac9b52","Type":"ContainerDied","Data":"546303a23c4d2730478e14f596e7ca9561ee2e58eb63df3e605736acb1e2aa24"} Nov 02 23:39:40.663697 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.663680 5672 scope.go:122] "RemoveContainer" containerID="546303a23c4d2730478e14f596e7ca9561ee2e58eb63df3e605736acb1e2aa24" Nov 02 23:39:40.665206 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.665182 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:40.665271 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.665248 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:40.665307 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.665280 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:39:40.805032 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.804999 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7"] Nov 02 23:39:40.843744 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:39:40.843703 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11791814_60d7_405a_ae75_ff0b9e96983a.slice/crio-5a769d4ac7ef0da8daea7d35d98cdecff9c3ecef3ef94c076fc3b5109282890e WatchSource:0}: Error finding container 5a769d4ac7ef0da8daea7d35d98cdecff9c3ecef3ef94c076fc3b5109282890e: Status 404 returned error can't find the container with id 5a769d4ac7ef0da8daea7d35d98cdecff9c3ecef3ef94c076fc3b5109282890e Nov 02 23:39:40.989327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:40.989301 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf"] Nov 02 23:39:40.992325 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:39:40.992296 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b9f6bc_bcbb_4eec_b5c0_34aadfe1f10b.slice/crio-2b7798b09f599a61dbd098e2d0266459b78011b6dbfd63d2d2945b88e8b71282 WatchSource:0}: Error finding container 2b7798b09f599a61dbd098e2d0266459b78011b6dbfd63d2d2945b88e8b71282: Status 404 returned error can't find the container with id 2b7798b09f599a61dbd098e2d0266459b78011b6dbfd63d2d2945b88e8b71282 Nov 02 23:39:41.671524 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.671484 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" event={"ID":"11791814-60d7-405a-ae75-ff0b9e96983a","Type":"ContainerStarted","Data":"4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b"} Nov 02 23:39:41.672048 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.671550 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" event={"ID":"11791814-60d7-405a-ae75-ff0b9e96983a","Type":"ContainerStarted","Data":"5a769d4ac7ef0da8daea7d35d98cdecff9c3ecef3ef94c076fc3b5109282890e"} Nov 02 23:39:41.672048 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.671695 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 02 23:39:41.673750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.673727 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" event={"ID":"9adcfc0e-353b-44e2-b6ee-8a915fac9b52","Type":"ContainerStarted","Data":"4edf1262ad260f9ebbb9cafd4db0a12ed607992a6870a215d574bcc5ff74b3f2"} Nov 02 23:39:41.675360 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.675336 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" event={"ID":"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b","Type":"ContainerStarted","Data":"60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba"} Nov 02 23:39:41.675492 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.675387 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 02 23:39:41.675492 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.675413 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" event={"ID":"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b","Type":"ContainerStarted","Data":"2b7798b09f599a61dbd098e2d0266459b78011b6dbfd63d2d2945b88e8b71282"} Nov 02 23:39:41.775863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.775822 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:39:41.776297 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.776273 5672 patch_prober.go:29] interesting pod/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 02 23:39:41.776381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.776342 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 02 23:39:41.779944 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.779897 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" podStartSLOduration=1.779885707 podStartE2EDuration="1.779885707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:39:41.776175326 +0000 UTC m=+2775.183487149" watchObservedRunningTime="2026-11-02 23:39:41.779885707 +0000 UTC m=+2775.187197534" Nov 02 23:39:41.999098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:41.999044 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" podStartSLOduration=1.9990293559999999 podStartE2EDuration="1.999029356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:39:41.995319118 +0000 UTC m=+2775.402630949" watchObservedRunningTime="2026-11-02 23:39:41.999029356 +0000 UTC m=+2775.406341186" Nov 02 23:39:42.776457 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:42.776428 5672 patch_prober.go:29] interesting pod/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 02 23:39:42.776844 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:42.776479 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 02 23:39:43.776577 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:43.776532 5672 patch_prober.go:29] interesting pod/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 02 23:39:43.777010 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:43.776605 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 02 23:39:44.775858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:44.775819 5672 patch_prober.go:29] interesting pod/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 02 23:39:44.776018 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:44.775881 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 02 23:39:45.776399 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:45.776357 5672 patch_prober.go:29] interesting pod/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Nov 02 23:39:45.776810 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:45.776441 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Nov 02 23:39:46.828186 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:46.828128 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:39:46.829191 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:46.829115 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:39:46.830352 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:46.830320 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 02 23:39:52.684837 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:52.684806 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 02 23:39:52.685258 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:52.684945 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 02 23:39:53.044648 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:53.044614 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw"] Nov 02 23:39:53.047636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:53.047609 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1d8584-hgvcw"] Nov 02 23:39:54.562354 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.562308 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj"] Nov 02 23:39:54.567010 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.566986 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 02 23:39:54.617515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.617486 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj"] Nov 02 23:39:54.672498 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.672462 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-c25940\"" Nov 02 23:39:54.673221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.673192 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-c25940-dockercfg-pnfxn\"" Nov 02 23:39:54.698103 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.697822 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e02f1e-bd39-416b-9323-5eed40f53cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj\" (UID: \"49e02f1e-bd39-416b-9323-5eed40f53cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 02 23:39:54.698103 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.697931 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgm82\" (UniqueName: \"kubernetes.io/projected/49e02f1e-bd39-416b-9323-5eed40f53cb3-kube-api-access-tgm82\") pod \"payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj\" (UID: \"49e02f1e-bd39-416b-9323-5eed40f53cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 02 23:39:54.739003 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.738964 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebaef9a0-d3f6-4953-a457-13d84409ffcc" path="/var/lib/kubelet/pods/ebaef9a0-d3f6-4953-a457-13d84409ffcc/volumes" Nov 02 23:39:54.739354 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.739335 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw"] Nov 02 23:39:54.743946 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.743322 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 02 23:39:54.767576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.767193 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw"] Nov 02 23:39:54.799465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.799431 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85zc\" (UniqueName: \"kubernetes.io/projected/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-kube-api-access-q85zc\") pod \"payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw\" (UID: \"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 02 23:39:54.799630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.799486 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e02f1e-bd39-416b-9323-5eed40f53cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj\" (UID: \"49e02f1e-bd39-416b-9323-5eed40f53cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 02 23:39:54.799630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.799619 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgm82\" (UniqueName: \"kubernetes.io/projected/49e02f1e-bd39-416b-9323-5eed40f53cb3-kube-api-access-tgm82\") pod \"payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj\" (UID: \"49e02f1e-bd39-416b-9323-5eed40f53cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 02 23:39:54.799825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.799802 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw\" (UID: \"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 02 23:39:54.800272 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.800251 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e02f1e-bd39-416b-9323-5eed40f53cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj\" (UID: \"49e02f1e-bd39-416b-9323-5eed40f53cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 02 23:39:54.853793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.853734 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgm82\" (UniqueName: \"kubernetes.io/projected/49e02f1e-bd39-416b-9323-5eed40f53cb3-kube-api-access-tgm82\") pod \"payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj\" (UID: \"49e02f1e-bd39-416b-9323-5eed40f53cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 02 23:39:54.881321 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.881292 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 02 23:39:54.900682 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.900657 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw\" (UID: \"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 02 23:39:54.900801 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.900708 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85zc\" (UniqueName: \"kubernetes.io/projected/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-kube-api-access-q85zc\") pod \"payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw\" (UID: \"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 02 23:39:54.901365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.901344 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw\" (UID: \"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 02 23:39:54.956301 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:54.956266 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85zc\" (UniqueName: \"kubernetes.io/projected/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-kube-api-access-q85zc\") pod \"payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw\" (UID: \"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 02 23:39:55.046546 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.046501 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk"] Nov 02 23:39:55.051449 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.051420 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8343b3f2-65hvk"] Nov 02 23:39:55.059788 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.059763 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 02 23:39:55.271470 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.271437 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj"] Nov 02 23:39:55.272735 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:39:55.272706 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e02f1e_bd39_416b_9323_5eed40f53cb3.slice/crio-d2cdf8d33bbe636dd19933c4394f485e7cc2b63bf9d2953696dfe5f43930b68c WatchSource:0}: Error finding container d2cdf8d33bbe636dd19933c4394f485e7cc2b63bf9d2953696dfe5f43930b68c: Status 404 returned error can't find the container with id d2cdf8d33bbe636dd19933c4394f485e7cc2b63bf9d2953696dfe5f43930b68c Nov 02 23:39:55.430945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.430917 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw"] Nov 02 23:39:55.431700 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:39:55.431668 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efd45fd_b21e_4ca7_b2d7_88f15ae1d168.slice/crio-116c53299656899ab36d1fba69dbf19484bbcfacdaa933da97ad5b76d8a74b4f WatchSource:0}: Error finding container 116c53299656899ab36d1fba69dbf19484bbcfacdaa933da97ad5b76d8a74b4f: Status 404 returned error can't find the container with id 116c53299656899ab36d1fba69dbf19484bbcfacdaa933da97ad5b76d8a74b4f Nov 02 23:39:55.790124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.790090 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" event={"ID":"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168","Type":"ContainerStarted","Data":"bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b"} Nov 02 23:39:55.790684 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.790143 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" event={"ID":"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168","Type":"ContainerStarted","Data":"116c53299656899ab36d1fba69dbf19484bbcfacdaa933da97ad5b76d8a74b4f"} Nov 02 23:39:55.790684 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.790198 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 02 23:39:55.791806 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.791781 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" event={"ID":"49e02f1e-bd39-416b-9323-5eed40f53cb3","Type":"ContainerStarted","Data":"5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db"} Nov 02 23:39:55.791924 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.791820 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" event={"ID":"49e02f1e-bd39-416b-9323-5eed40f53cb3","Type":"ContainerStarted","Data":"d2cdf8d33bbe636dd19933c4394f485e7cc2b63bf9d2953696dfe5f43930b68c"} Nov 02 23:39:55.791924 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.791860 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 02 23:39:55.900861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:55.900808 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" podStartSLOduration=1.90079552 podStartE2EDuration="1.90079552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:39:55.897264203 +0000 UTC m=+2789.304576031" watchObservedRunningTime="2026-11-02 23:39:55.90079552 +0000 UTC m=+2789.308107383" Nov 02 23:39:56.003483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:56.003424 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" podStartSLOduration=2.003409749 podStartE2EDuration="2.003409749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:39:56.002991556 +0000 UTC m=+2789.410303389" watchObservedRunningTime="2026-11-02 23:39:56.003409749 +0000 UTC m=+2789.410721579" Nov 02 23:39:56.026932 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:39:56.026841 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:39:56.713214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:39:56.713175 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc6a834-a50c-474b-81de-61212576468c" path="/var/lib/kubelet/pods/3fc6a834-a50c-474b-81de-61212576468c/volumes" Nov 02 23:40:06.803263 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:06.803179 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 02 23:40:06.803868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:06.803487 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 02 23:40:08.834336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.834301 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg"] Nov 02 23:40:08.839703 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.839660 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:08.901418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.898745 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7ca60f-openshift-default-dockercfg-bp4zd\"" Nov 02 23:40:08.975364 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.965006 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg"] Nov 02 23:40:08.975554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.966584 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-certs\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:08.975554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.975499 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e415f6-4842-4991-bd47-03d50d0354bf-istiod-ca-cert\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:08.975554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.975534 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m446m\" (UniqueName: \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-kube-api-access-m446m\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:08.975741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.975577 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-data\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:08.975741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.975616 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-envoy\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:08.975850 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.975809 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-credential-socket\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:08.975910 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.975882 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-istio-token\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:08.975963 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.975924 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e415f6-4842-4991-bd47-03d50d0354bf-istio-podinfo\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:08.976090 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:08.976065 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-socket\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.077054 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.077009 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-certs\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.077299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.077063 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e415f6-4842-4991-bd47-03d50d0354bf-istiod-ca-cert\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.077299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.077092 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m446m\" (UniqueName: \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-kube-api-access-m446m\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.077299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.077119 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-data\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.077299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.077181 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-envoy\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.077299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.077264 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-credential-socket\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.077842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.077320 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-istio-token\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.077842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.077353 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e415f6-4842-4991-bd47-03d50d0354bf-istio-podinfo\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.077842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.077399 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-socket\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.077842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.077503 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-certs\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.077842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.077734 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-credential-socket\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.078576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.078328 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-data\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.078725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.078683 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-socket\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.079053 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.079030 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e415f6-4842-4991-bd47-03d50d0354bf-istiod-ca-cert\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.079411 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.079387 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz"] Nov 02 23:40:09.082524 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.082500 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-envoy\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.082850 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.082802 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e415f6-4842-4991-bd47-03d50d0354bf-istio-podinfo\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.094595 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.094542 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62f45f2-pspxz"] Nov 02 23:40:09.132397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.132350 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m446m\" (UniqueName: \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-kube-api-access-m446m\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.132818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.132799 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-istio-token\") pod \"e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.155027 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.154998 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:09.575959 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.575933 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg"] Nov 02 23:40:09.577762 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:40:09.577733 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e415f6_4842_4991_bd47_03d50d0354bf.slice/crio-213b3c3e4eda179e0c2cd3f723dd493dc6711a073e08de0cd4140b867735b0dd WatchSource:0}: Error finding container 213b3c3e4eda179e0c2cd3f723dd493dc6711a073e08de0cd4140b867735b0dd: Status 404 returned error can't find the container with id 213b3c3e4eda179e0c2cd3f723dd493dc6711a073e08de0cd4140b867735b0dd Nov 02 23:40:09.580592 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.580556 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:40:09.580703 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.580633 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:40:09.580703 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.580676 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:40:09.980063 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.980031 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" event={"ID":"33e415f6-4842-4991-bd47-03d50d0354bf","Type":"ContainerStarted","Data":"aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90"} Nov 02 23:40:09.980489 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:09.980078 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" event={"ID":"33e415f6-4842-4991-bd47-03d50d0354bf","Type":"ContainerStarted","Data":"213b3c3e4eda179e0c2cd3f723dd493dc6711a073e08de0cd4140b867735b0dd"} Nov 02 23:40:10.091075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:10.091023 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" podStartSLOduration=2.091008864 podStartE2EDuration="2.091008864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:40:10.08957026 +0000 UTC m=+2803.496882089" watchObservedRunningTime="2026-11-02 23:40:10.091008864 +0000 UTC m=+2803.498320694" Nov 02 23:40:10.156193 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:10.156134 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:10.161495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:10.161471 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:10.715937 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:10.715901 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34daf46b-0833-45a6-981b-7b7c545be946" path="/var/lib/kubelet/pods/34daf46b-0833-45a6-981b-7b7c545be946/volumes" Nov 02 23:40:10.987111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:10.987082 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:10.987971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:10.987949 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:40:12.900194 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:12.900130 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh"] Nov 02 23:40:12.904177 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:12.904135 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 02 23:40:12.914914 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:12.914889 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612b0292-9195-4729-bcfb-44ea8d80a9c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh\" (UID: \"612b0292-9195-4729-bcfb-44ea8d80a9c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 02 23:40:12.915036 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:12.915005 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9wgv\" (UniqueName: \"kubernetes.io/projected/612b0292-9195-4729-bcfb-44ea8d80a9c0-kube-api-access-b9wgv\") pod \"payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh\" (UID: \"612b0292-9195-4729-bcfb-44ea8d80a9c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 02 23:40:12.967256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:12.967222 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh"] Nov 02 23:40:13.002538 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:13.002509 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-a51378\"" Nov 02 23:40:13.002698 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:13.002564 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-a51378-dockercfg-kllfp\"" Nov 02 23:40:13.015695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:13.015665 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612b0292-9195-4729-bcfb-44ea8d80a9c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh\" (UID: \"612b0292-9195-4729-bcfb-44ea8d80a9c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 02 23:40:13.015846 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:13.015767 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9wgv\" (UniqueName: \"kubernetes.io/projected/612b0292-9195-4729-bcfb-44ea8d80a9c0-kube-api-access-b9wgv\") pod \"payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh\" (UID: \"612b0292-9195-4729-bcfb-44ea8d80a9c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 02 23:40:13.016519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:13.016495 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612b0292-9195-4729-bcfb-44ea8d80a9c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh\" (UID: \"612b0292-9195-4729-bcfb-44ea8d80a9c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 02 23:40:13.071818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:13.071785 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9wgv\" (UniqueName: \"kubernetes.io/projected/612b0292-9195-4729-bcfb-44ea8d80a9c0-kube-api-access-b9wgv\") pod \"payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh\" (UID: \"612b0292-9195-4729-bcfb-44ea8d80a9c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 02 23:40:13.218593 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:13.218566 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 02 23:40:13.601443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:13.601414 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh"] Nov 02 23:40:13.602380 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:40:13.602329 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612b0292_9195_4729_bcfb_44ea8d80a9c0.slice/crio-e1d9641bb7031cffe2296c2e23fef9867eb566951f9f203b4acafd571080d69e WatchSource:0}: Error finding container e1d9641bb7031cffe2296c2e23fef9867eb566951f9f203b4acafd571080d69e: Status 404 returned error can't find the container with id e1d9641bb7031cffe2296c2e23fef9867eb566951f9f203b4acafd571080d69e Nov 02 23:40:14.012135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:14.012100 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" event={"ID":"612b0292-9195-4729-bcfb-44ea8d80a9c0","Type":"ContainerStarted","Data":"82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782"} Nov 02 23:40:14.012583 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:14.012173 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" event={"ID":"612b0292-9195-4729-bcfb-44ea8d80a9c0","Type":"ContainerStarted","Data":"e1d9641bb7031cffe2296c2e23fef9867eb566951f9f203b4acafd571080d69e"} Nov 02 23:40:14.012583 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:14.012333 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 02 23:40:14.119071 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:14.119005 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" podStartSLOduration=2.118987638 podStartE2EDuration="2.118987638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:40:14.118722805 +0000 UTC m=+2807.526034626" watchObservedRunningTime="2026-11-02 23:40:14.118987638 +0000 UTC m=+2807.526299470" Nov 02 23:40:19.670256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.670221 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh"] Nov 02 23:40:19.675859 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.675835 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 02 23:40:19.711355 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.711325 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh\" (UID: \"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 02 23:40:19.711486 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.711466 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm45q\" (UniqueName: \"kubernetes.io/projected/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-kube-api-access-zm45q\") pod \"payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh\" (UID: \"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 02 23:40:19.771902 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.771848 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7ca60f\"" Nov 02 23:40:19.772089 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.771906 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7ca60f-dockercfg-sxtbb\"" Nov 02 23:40:19.776371 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.776343 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh"] Nov 02 23:40:19.812647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.812621 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh\" (UID: \"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 02 23:40:19.812796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.812723 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm45q\" (UniqueName: \"kubernetes.io/projected/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-kube-api-access-zm45q\") pod \"payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh\" (UID: \"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 02 23:40:19.813371 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.813352 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh\" (UID: \"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 02 23:40:19.865937 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.865910 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm45q\" (UniqueName: \"kubernetes.io/projected/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-kube-api-access-zm45q\") pod \"payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh\" (UID: \"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 02 23:40:19.989041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:19.989014 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 02 23:40:20.360230 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:40:20.360193 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95cee42_e848_4f4b_98e5_c4c9c2c00c3a.slice/crio-5c88b8e09f4e537827cc48258bd3327c7a4ebae318abaed06b5e30a32bc51261 WatchSource:0}: Error finding container 5c88b8e09f4e537827cc48258bd3327c7a4ebae318abaed06b5e30a32bc51261: Status 404 returned error can't find the container with id 5c88b8e09f4e537827cc48258bd3327c7a4ebae318abaed06b5e30a32bc51261 Nov 02 23:40:20.361031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:20.361004 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh"] Nov 02 23:40:21.090803 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:21.090767 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" event={"ID":"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a","Type":"ContainerStarted","Data":"47da314c6eaf4bd79da7b66c959f7d2ba0ef8ac01682281095c2427036ab7352"} Nov 02 23:40:21.091243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:21.090824 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" event={"ID":"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a","Type":"ContainerStarted","Data":"5c88b8e09f4e537827cc48258bd3327c7a4ebae318abaed06b5e30a32bc51261"} Nov 02 23:40:21.091243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:21.090854 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 02 23:40:21.199558 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:21.199501 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" podStartSLOduration=2.19948762 podStartE2EDuration="2.19948762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:40:21.198445442 +0000 UTC m=+2814.605757283" watchObservedRunningTime="2026-11-02 23:40:21.19948762 +0000 UTC m=+2814.606799450" Nov 02 23:40:25.021434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:25.021406 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 02 23:40:26.035794 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:40:26.035745 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:40:26.046075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:26.045698 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8"] Nov 02 23:40:26.090424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:26.090359 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6dada84-cdvz8"] Nov 02 23:40:26.713051 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:26.713016 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cbfd6f9-581a-4136-86a8-f5ec01c7454b" path="/var/lib/kubelet/pods/9cbfd6f9-581a-4136-86a8-f5ec01c7454b/volumes" Nov 02 23:40:28.450926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:28.450897 5672 scope.go:122] "RemoveContainer" containerID="1ea1dc14260f9285314ab10abb2cd51c23487ab04fe456402f6647569c1ea7f8" Nov 02 23:40:28.460845 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:28.460827 5672 scope.go:122] "RemoveContainer" containerID="a8f85eb9298a95a87560c4a44bfb3eb7a169cd74aa3e6513d9da7017c472dbb6" Nov 02 23:40:28.471212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:28.470597 5672 scope.go:122] "RemoveContainer" containerID="c22135e6eba8bc7a8b6ab1a1607d050a20d93e26c3ddf0938f566f604ed849e7" Nov 02 23:40:28.482387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:28.482340 5672 scope.go:122] "RemoveContainer" containerID="be64fddd05233486afa0faf5fc89dc5be0fb1fd0b0ccd482241383ab4518f0ea" Nov 02 23:40:28.490821 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:28.490796 5672 scope.go:122] "RemoveContainer" containerID="366e5239c0375ca8e80c0c93bf24205d9758557fa636b5846968ae6c2bee2105" Nov 02 23:40:32.101424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:32.101396 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 02 23:40:53.033089 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:40:53.033056 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:40:56.021644 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:40:56.021609 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:41:04.434275 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:04.434239 5672 generic.go:356] "Generic (PLEG): container finished" podID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerID="ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b" exitCode=0 Nov 02 23:41:04.434863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:04.434286 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" event={"ID":"7ed3e348-d810-43a0-afa2-cf8083bc8874","Type":"ContainerDied","Data":"ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b"} Nov 02 23:41:04.434863 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:04.434736 5672 scope.go:122] "RemoveContainer" containerID="ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b" Nov 02 23:41:04.436725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:04.436687 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:41:04.436836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:04.436774 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:41:04.436836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:04.436816 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:41:05.445256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:05.445223 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" event={"ID":"7ed3e348-d810-43a0-afa2-cf8083bc8874","Type":"ContainerStarted","Data":"2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110"} Nov 02 23:41:05.608972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:05.608943 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:41:05.799537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:05.799502 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:41:05.800633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:05.800597 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 23:41:05.800796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:05.800646 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 23:41:06.800105 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:06.800067 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 23:41:06.800528 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:06.800116 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 23:41:07.800080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:07.800043 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 23:41:07.800294 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:07.800111 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 23:41:08.799517 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:08.799480 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 23:41:08.799715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:08.799536 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 23:41:09.799677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:09.799639 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 23:41:09.800078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:09.799721 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 23:41:10.799479 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:10.799434 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 23:41:10.799712 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:10.799503 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 23:41:11.804119 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:11.804090 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:41:11.805008 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:11.804990 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 02 23:41:21.052616 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:21.052582 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg"] Nov 02 23:41:21.098465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:21.098434 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8fd099-2bpmg"] Nov 02 23:41:22.712384 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:22.712354 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84d7488-bd24-4afa-bf9a-a79776fbb3b4" path="/var/lib/kubelet/pods/c84d7488-bd24-4afa-bf9a-a79776fbb3b4/volumes" Nov 02 23:41:26.021257 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:41:26.021223 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:41:28.537218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:28.537178 5672 scope.go:122] "RemoveContainer" containerID="dfbedee0d8cdd96c677b0414a63cd2ff1f50002308ae8fb458511779b00d7023" Nov 02 23:41:28.548448 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:28.548424 5672 scope.go:122] "RemoveContainer" containerID="6d9ef55be6ae6872df7560822c90b2997cc912950f6a58d63309a5eac03121c6" Nov 02 23:41:28.559427 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:28.559402 5672 scope.go:122] "RemoveContainer" containerID="72657f7d516f9e0de2ebff901eefbba6feda675f8160e6e611ac70ffdff188dc" Nov 02 23:41:28.570341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:28.570310 5672 scope.go:122] "RemoveContainer" containerID="d6581234bd8d76a73f2c760fd9ee8fcc6569e79ae0307d2ec15c83cd7708eb45" Nov 02 23:41:28.579282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:28.579261 5672 scope.go:122] "RemoveContainer" containerID="586a3642c50e54eafa7bd86da1ace6736883771fcc379937419c60545471cd62" Nov 02 23:41:28.588256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:28.588236 5672 scope.go:122] "RemoveContainer" containerID="2de2ce30b44bc8c9e8ddc1313cd638626512262e57e9d9ea848e1a6602733579" Nov 02 23:41:30.613052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:30.613019 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:41:36.045334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:36.045246 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq"] Nov 02 23:41:36.091363 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:36.091330 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3a6c40-v7gwq"] Nov 02 23:41:36.712975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:36.712943 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c12035-8647-4b3a-bb55-f5338cbe0fee" path="/var/lib/kubelet/pods/c5c12035-8647-4b3a-bb55-f5338cbe0fee/volumes" Nov 02 23:41:40.753444 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:40.753411 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv"] Nov 02 23:41:45.754686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:45.754645 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" podUID="68e86d19-a8c6-44b0-9076-f47f4221b525" containerName="payload-pre-processing" containerID="cri-o://f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92" gracePeriod=25 Nov 02 23:41:46.021434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.021371 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:41:46.067590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.067552 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68e86d19-a8c6-44b0-9076-f47f4221b525-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e86d19-a8c6-44b0-9076-f47f4221b525-plugins-config-volume\") pod \"68e86d19-a8c6-44b0-9076-f47f4221b525\" (UID: \"68e86d19-a8c6-44b0-9076-f47f4221b525\") " Nov 02 23:41:46.067775 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.067757 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e86d19-a8c6-44b0-9076-f47f4221b525-kube-api-access-tt7mq\" (UniqueName: \"kubernetes.io/projected/68e86d19-a8c6-44b0-9076-f47f4221b525-kube-api-access-tt7mq\") pod \"68e86d19-a8c6-44b0-9076-f47f4221b525\" (UID: \"68e86d19-a8c6-44b0-9076-f47f4221b525\") " Nov 02 23:41:46.067920 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.067894 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e86d19-a8c6-44b0-9076-f47f4221b525-plugins-config-volume" pod "68e86d19-a8c6-44b0-9076-f47f4221b525" (UID: "68e86d19-a8c6-44b0-9076-f47f4221b525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:41:46.068274 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.068254 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e86d19-a8c6-44b0-9076-f47f4221b525-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:46.069916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.069894 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e86d19-a8c6-44b0-9076-f47f4221b525-kube-api-access-tt7mq" pod "68e86d19-a8c6-44b0-9076-f47f4221b525" (UID: "68e86d19-a8c6-44b0-9076-f47f4221b525"). InnerVolumeSpecName "kube-api-access-tt7mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:41:46.169724 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.169689 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt7mq\" (UniqueName: \"kubernetes.io/projected/68e86d19-a8c6-44b0-9076-f47f4221b525-kube-api-access-tt7mq\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:41:46.767586 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.767552 5672 generic.go:356] "Generic (PLEG): container finished" podID="68e86d19-a8c6-44b0-9076-f47f4221b525" containerID="f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92" exitCode=0 Nov 02 23:41:46.768027 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.767596 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" Nov 02 23:41:46.768027 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.767631 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" event={"ID":"68e86d19-a8c6-44b0-9076-f47f4221b525","Type":"ContainerDied","Data":"f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92"} Nov 02 23:41:46.768027 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.767683 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv" event={"ID":"68e86d19-a8c6-44b0-9076-f47f4221b525","Type":"ContainerDied","Data":"ca33d360be4b7427cac2af92028d1ad60ff02b3017c555f21dcc9be4619ed26c"} Nov 02 23:41:46.768027 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.767705 5672 scope.go:122] "RemoveContainer" containerID="f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92" Nov 02 23:41:46.777722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.777705 5672 scope.go:122] "RemoveContainer" containerID="f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92" Nov 02 23:41:46.778014 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:41:46.777992 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92\": container with ID starting with f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92 not found: ID does not exist" containerID="f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92" Nov 02 23:41:46.778061 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.778026 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92"} err="failed to get container status \"f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92\": rpc error: code = NotFound desc = could not find container \"f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92\": container with ID starting with f15c57e1a1597e7cb0dcd05bf1095a99c2ffd790262f3185ac2ae4eaf9afaa92 not found: ID does not exist" Nov 02 23:41:46.924858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.924817 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv"] Nov 02 23:41:46.928173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:46.928130 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b1be95-8667ff4cb7-nwzmv"] Nov 02 23:41:48.712587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:48.712553 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e86d19-a8c6-44b0-9076-f47f4221b525" path="/var/lib/kubelet/pods/68e86d19-a8c6-44b0-9076-f47f4221b525/volumes" Nov 02 23:41:52.045102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:52.045056 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2"] Nov 02 23:41:52.091406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:52.091374 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1807fa51-zb7m2"] Nov 02 23:41:52.712342 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:41:52.712314 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f86f02-c5ac-45a7-b39f-2a13973ac864" path="/var/lib/kubelet/pods/39f86f02-c5ac-45a7-b39f-2a13973ac864/volumes" Nov 02 23:41:56.020469 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:41:56.020432 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:42:07.047399 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:07.047361 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c"] Nov 02 23:42:07.093659 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:07.093614 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30fb1af1-drq6c"] Nov 02 23:42:07.961400 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:07.961368 5672 generic.go:356] "Generic (PLEG): container finished" podID="33e415f6-4842-4991-bd47-03d50d0354bf" containerID="aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90" exitCode=0 Nov 02 23:42:07.961656 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:07.961447 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" event={"ID":"33e415f6-4842-4991-bd47-03d50d0354bf","Type":"ContainerDied","Data":"aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90"} Nov 02 23:42:07.961934 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:07.961916 5672 scope.go:122] "RemoveContainer" containerID="aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90" Nov 02 23:42:07.963584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:07.963554 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:42:07.963672 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:07.963627 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:42:07.963672 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:07.963662 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:42:08.717328 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:08.717299 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c207c04-3728-4bf2-98f6-091f61976ba4" path="/var/lib/kubelet/pods/1c207c04-3728-4bf2-98f6-091f61976ba4/volumes" Nov 02 23:42:08.972428 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:08.972392 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" event={"ID":"33e415f6-4842-4991-bd47-03d50d0354bf","Type":"ContainerStarted","Data":"92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009"} Nov 02 23:42:09.073309 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:09.073271 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6"] Nov 02 23:42:09.078157 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:09.078107 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4447eb-vllt6"] Nov 02 23:42:09.155343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:09.155301 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:42:09.156579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:09.156544 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 02 23:42:09.156728 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:09.156612 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 02 23:42:10.155650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:10.155498 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 02 23:42:10.155650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:10.155558 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 02 23:42:10.714612 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:10.714581 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca7a002-ac54-4415-8fd2-b923419dcfa4" path="/var/lib/kubelet/pods/1ca7a002-ac54-4415-8fd2-b923419dcfa4/volumes" Nov 02 23:42:10.987188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:10.987137 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:42:11.155533 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:11.155483 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 02 23:42:11.155738 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:11.155543 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 02 23:42:12.155483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:12.155445 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 02 23:42:12.155722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:12.155515 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 02 23:42:13.159178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:13.157525 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 02 23:42:13.159178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:13.157588 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 02 23:42:14.155614 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:14.155573 5672 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 02 23:42:14.155848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:14.155641 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 02 23:42:15.161527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:15.161490 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:42:15.163645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:15.163618 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 02 23:42:15.941838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:15.941800 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:42:18.044831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:18.044797 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx"] Nov 02 23:42:18.091014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:18.090982 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56a829f7-mjrwx"] Nov 02 23:42:18.713176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:18.713129 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59900d47-d15a-493a-800b-88bead4da834" path="/var/lib/kubelet/pods/59900d47-d15a-493a-800b-88bead4da834/volumes" Nov 02 23:42:26.021334 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:42:26.021298 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:42:28.046216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:28.046171 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg"] Nov 02 23:42:28.092403 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:28.092375 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5293a438-fx7wg"] Nov 02 23:42:28.637375 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:28.637343 5672 scope.go:122] "RemoveContainer" containerID="0b9478f4c978f87829581889ad81e5427cba10f201d57dd894e05590181a5cee" Nov 02 23:42:28.649506 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:28.649482 5672 scope.go:122] "RemoveContainer" containerID="f6ddb9e66b43e39b9f566bcd27603f88990bbf744f9186b62a21b5a1c881d26d" Nov 02 23:42:28.712576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:28.712542 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99bbc390-437d-4e6c-acf5-2853dc335d03" path="/var/lib/kubelet/pods/99bbc390-437d-4e6c-acf5-2853dc335d03/volumes" Nov 02 23:42:32.044771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:32.044736 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4"] Nov 02 23:42:32.090941 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:32.090910 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09d399be-btjm4"] Nov 02 23:42:32.713335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:32.713302 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989cbadc-d89d-4c78-859d-fa0f0eecaa31" path="/var/lib/kubelet/pods/989cbadc-d89d-4c78-859d-fa0f0eecaa31/volumes" Nov 02 23:42:33.043465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:33.043436 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs"] Nov 02 23:42:33.090459 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:33.090427 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4a3413-m25rs"] Nov 02 23:42:34.712468 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:34.712435 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dbeaaa0-4be3-4be2-afa2-87604003d7f8" path="/var/lib/kubelet/pods/2dbeaaa0-4be3-4be2-afa2-87604003d7f8/volumes" Nov 02 23:42:43.045954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:43.045922 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz"] Nov 02 23:42:43.092065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:43.092016 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42665ce8-m4jpz"] Nov 02 23:42:44.712647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:44.712612 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d622289-ec5c-4d0a-a380-f5d077ed6b17" path="/var/lib/kubelet/pods/8d622289-ec5c-4d0a-a380-f5d077ed6b17/volumes" Nov 02 23:42:48.044768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:48.044724 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl"] Nov 02 23:42:48.089343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:48.089313 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c8781bb-nkjtl"] Nov 02 23:42:48.712713 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:48.712682 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f0ef24-c7ae-4d9e-977d-03de174928d3" path="/var/lib/kubelet/pods/20f0ef24-c7ae-4d9e-977d-03de174928d3/volumes" Nov 02 23:42:49.045743 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:49.045704 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77"] Nov 02 23:42:49.090549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:49.090515 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33dea87-rcd77"] Nov 02 23:42:50.128381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:50.128347 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:42:50.713308 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:42:50.713266 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf263724-6e40-4a97-abc5-3c8cdff3ff2f" path="/var/lib/kubelet/pods/cf263724-6e40-4a97-abc5-3c8cdff3ff2f/volumes" Nov 02 23:42:56.021897 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:42:56.021853 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:43:20.537259 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.537042 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g"] Nov 02 23:43:20.537883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.537861 5672 cpu_manager.go:414] "containerMap: removing container" podUID="68e86d19-a8c6-44b0-9076-f47f4221b525" containerName="payload-pre-processing" Nov 02 23:43:20.537978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.537887 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e86d19-a8c6-44b0-9076-f47f4221b525" containerName="payload-pre-processing" Nov 02 23:43:20.538057 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.538043 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e86d19-a8c6-44b0-9076-f47f4221b525" containerName="payload-pre-processing" Nov 02 23:43:20.541928 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.541904 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:43:20.593511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.593473 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g"] Nov 02 23:43:20.624311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.624268 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957cd081-e9dc-42ea-8376-551baa2df4be-plugins-config-volume\") pod \"payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g\" (UID: \"957cd081-e9dc-42ea-8376-551baa2df4be\") " pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:43:20.624499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.624362 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd64x\" (UniqueName: \"kubernetes.io/projected/957cd081-e9dc-42ea-8376-551baa2df4be-kube-api-access-qd64x\") pod \"payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g\" (UID: \"957cd081-e9dc-42ea-8376-551baa2df4be\") " pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:43:20.636690 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.636650 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-52dd59\"" Nov 02 23:43:20.636836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.636703 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-52dd59-dockercfg-m8mlw\"" Nov 02 23:43:20.726211 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.726170 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957cd081-e9dc-42ea-8376-551baa2df4be-plugins-config-volume\") pod \"payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g\" (UID: \"957cd081-e9dc-42ea-8376-551baa2df4be\") " pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:43:20.726462 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.726264 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd64x\" (UniqueName: \"kubernetes.io/projected/957cd081-e9dc-42ea-8376-551baa2df4be-kube-api-access-qd64x\") pod \"payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g\" (UID: \"957cd081-e9dc-42ea-8376-551baa2df4be\") " pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:43:20.727094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.727069 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957cd081-e9dc-42ea-8376-551baa2df4be-plugins-config-volume\") pod \"payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g\" (UID: \"957cd081-e9dc-42ea-8376-551baa2df4be\") " pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:43:20.779523 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.779494 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd64x\" (UniqueName: \"kubernetes.io/projected/957cd081-e9dc-42ea-8376-551baa2df4be-kube-api-access-qd64x\") pod \"payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g\" (UID: \"957cd081-e9dc-42ea-8376-551baa2df4be\") " pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:43:20.858015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:20.857922 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:43:21.232865 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:21.232821 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g"] Nov 02 23:43:21.234473 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:43:21.234441 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957cd081_e9dc_42ea_8376_551baa2df4be.slice/crio-536bd8c17e2d9959505c4ed2c47b1a6c0ffe6e98b3aeec971fd040d37f81eabe WatchSource:0}: Error finding container 536bd8c17e2d9959505c4ed2c47b1a6c0ffe6e98b3aeec971fd040d37f81eabe: Status 404 returned error can't find the container with id 536bd8c17e2d9959505c4ed2c47b1a6c0ffe6e98b3aeec971fd040d37f81eabe Nov 02 23:43:21.540437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:21.540332 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" event={"ID":"957cd081-e9dc-42ea-8376-551baa2df4be","Type":"ContainerStarted","Data":"1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a"} Nov 02 23:43:21.540437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:21.540402 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" event={"ID":"957cd081-e9dc-42ea-8376-551baa2df4be","Type":"ContainerStarted","Data":"536bd8c17e2d9959505c4ed2c47b1a6c0ffe6e98b3aeec971fd040d37f81eabe"} Nov 02 23:43:21.540437 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:21.540434 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:43:25.763554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:25.763495 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" podStartSLOduration=5.76347884 podStartE2EDuration="5.76347884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:43:21.648593007 +0000 UTC m=+2995.055904839" watchObservedRunningTime="2026-11-02 23:43:25.76347884 +0000 UTC m=+2999.170790670" Nov 02 23:43:25.764335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:25.764316 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:43:26.020907 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:43:26.020820 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:43:28.689575 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:28.689535 5672 scope.go:122] "RemoveContainer" containerID="e19f3aa7394db68408af2749796564ce267192adf3828d2b8aa8768c5cdcd965" Nov 02 23:43:28.701820 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:28.701796 5672 scope.go:122] "RemoveContainer" containerID="fe3154c7de6f3cafd355d357e58590ead3451edac1331d7ef0816f629cdf4d33" Nov 02 23:43:28.711044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:28.711022 5672 scope.go:122] "RemoveContainer" containerID="fa52e786ca94aec720c0f6d8ad4cf4db6585158f09afd8813aa578a53a136a73" Nov 02 23:43:28.721838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:28.721817 5672 scope.go:122] "RemoveContainer" containerID="afb21df32bbb75c3e28419292bfda2035fcef03e00cb482f1b163ec134e2731c" Nov 02 23:43:28.731787 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:28.731765 5672 scope.go:122] "RemoveContainer" containerID="4a2ec73bb2bfc77570d455524d9bea3d4ce7742ae1efb95fae252e82410a58f3" Nov 02 23:43:28.740306 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:28.740289 5672 scope.go:122] "RemoveContainer" containerID="643d7e226c2452e8bc4d37242ed2fc10e077f9293645541189916cddd4540300" Nov 02 23:43:32.550883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:32.550850 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:43:39.018476 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:39.018442 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh"] Nov 02 23:43:39.018891 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:39.018626 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" containerID="cri-o://bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101" gracePeriod=30 Nov 02 23:43:43.570093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:43.570052 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:43:43.570508 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:43.570134 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:43:44.294945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.294883 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:43:44.411195 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411137 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istiod-ca-cert\") pod \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " Nov 02 23:43:44.411387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411210 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-socket\") pod \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " Nov 02 23:43:44.411387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411277 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-data\") pod \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " Nov 02 23:43:44.411387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411314 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-certs\") pod \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " Nov 02 23:43:44.411387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411365 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-kube-api-access-wg96x\" (UniqueName: \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-kube-api-access-wg96x\") pod \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " Nov 02 23:43:44.411613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411438 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-podinfo\") pod \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " Nov 02 23:43:44.411613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411516 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-credential-socket\") pod \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " Nov 02 23:43:44.411613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411554 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-envoy\") pod \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " Nov 02 23:43:44.411613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411580 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istiod-ca-cert" pod "afabbfc9-a7bd-43f6-8923-83c13a54d7ac" (UID: "afabbfc9-a7bd-43f6-8923-83c13a54d7ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:44.411613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411583 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-data" pod "afabbfc9-a7bd-43f6-8923-83c13a54d7ac" (UID: "afabbfc9-a7bd-43f6-8923-83c13a54d7ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.411613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411591 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-token\" (UniqueName: \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-token\") pod \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\" (UID: \"afabbfc9-a7bd-43f6-8923-83c13a54d7ac\") " Nov 02 23:43:44.411966 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.411938 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-certs" pod "afabbfc9-a7bd-43f6-8923-83c13a54d7ac" (UID: "afabbfc9-a7bd-43f6-8923-83c13a54d7ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.412179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.412062 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-credential-socket" pod "afabbfc9-a7bd-43f6-8923-83c13a54d7ac" (UID: "afabbfc9-a7bd-43f6-8923-83c13a54d7ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.412179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.412106 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-socket" pod "afabbfc9-a7bd-43f6-8923-83c13a54d7ac" (UID: "afabbfc9-a7bd-43f6-8923-83c13a54d7ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.412311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.412263 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.412311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.412283 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.412311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.412301 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.412489 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.412315 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.412489 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.412330 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.413894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.413865 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-kube-api-access-wg96x" pod "afabbfc9-a7bd-43f6-8923-83c13a54d7ac" (UID: "afabbfc9-a7bd-43f6-8923-83c13a54d7ac"). InnerVolumeSpecName "kube-api-access-wg96x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:44.414006 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.413871 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-podinfo" pod "afabbfc9-a7bd-43f6-8923-83c13a54d7ac" (UID: "afabbfc9-a7bd-43f6-8923-83c13a54d7ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:43:44.414006 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.413932 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-token" pod "afabbfc9-a7bd-43f6-8923-83c13a54d7ac" (UID: "afabbfc9-a7bd-43f6-8923-83c13a54d7ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:44.414006 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.413950 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-envoy" pod "afabbfc9-a7bd-43f6-8923-83c13a54d7ac" (UID: "afabbfc9-a7bd-43f6-8923-83c13a54d7ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:44.513861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.513824 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg96x\" (UniqueName: \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-kube-api-access-wg96x\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.513861 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.513864 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.514072 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.513879 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.514072 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.513898 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afabbfc9-a7bd-43f6-8923-83c13a54d7ac-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:44.697275 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.697196 5672 generic.go:356] "Generic (PLEG): container finished" podID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerID="bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101" exitCode=0 Nov 02 23:43:44.697275 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.697248 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" Nov 02 23:43:44.697721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.697272 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" event={"ID":"afabbfc9-a7bd-43f6-8923-83c13a54d7ac","Type":"ContainerDied","Data":"bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101"} Nov 02 23:43:44.697721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.697329 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh" event={"ID":"afabbfc9-a7bd-43f6-8923-83c13a54d7ac","Type":"ContainerDied","Data":"03c86e22898a7636cf7c87f31937d7a9697262ce7e118ee9baabe50a3e626c4b"} Nov 02 23:43:44.697721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.697357 5672 scope.go:122] "RemoveContainer" containerID="bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101" Nov 02 23:43:44.709092 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.709070 5672 scope.go:122] "RemoveContainer" containerID="bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101" Nov 02 23:43:44.709416 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:43:44.709395 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101\": container with ID starting with bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101 not found: ID does not exist" containerID="bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101" Nov 02 23:43:44.709484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.709436 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101"} err="failed to get container status \"bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101\": rpc error: code = NotFound desc = could not find container \"bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101\": container with ID starting with bd0dedb75d64a07f3f34abbf41629e2a7b39e3f96c304e65d90ad14f2eaca101 not found: ID does not exist" Nov 02 23:43:44.850930 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.850895 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh"] Nov 02 23:43:44.861135 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:44.861101 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-5tsfh"] Nov 02 23:43:46.713851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:46.713815 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" path="/var/lib/kubelet/pods/afabbfc9-a7bd-43f6-8923-83c13a54d7ac/volumes" Nov 02 23:43:54.309771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:54.309732 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9"] Nov 02 23:43:54.310531 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:54.309970 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" podUID="996905d6-94af-41c4-915c-e69dfc9688e2" containerName="istio-proxy" containerID="cri-o://5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e" gracePeriod=30 Nov 02 23:43:56.026805 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:43:56.026760 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:43:59.571966 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.571939 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:43:59.703668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.703568 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-data\") pod \"996905d6-94af-41c4-915c-e69dfc9688e2\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " Nov 02 23:43:59.703668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.703642 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-envoy\") pod \"996905d6-94af-41c4-915c-e69dfc9688e2\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " Nov 02 23:43:59.703916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.703709 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-kube-api-access-plvp2\" (UniqueName: \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-kube-api-access-plvp2\") pod \"996905d6-94af-41c4-915c-e69dfc9688e2\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " Nov 02 23:43:59.703916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.703739 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-credential-socket\") pod \"996905d6-94af-41c4-915c-e69dfc9688e2\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " Nov 02 23:43:59.703916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.703866 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996905d6-94af-41c4-915c-e69dfc9688e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996905d6-94af-41c4-915c-e69dfc9688e2-istiod-ca-cert\") pod \"996905d6-94af-41c4-915c-e69dfc9688e2\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " Nov 02 23:43:59.704080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.703917 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/996905d6-94af-41c4-915c-e69dfc9688e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996905d6-94af-41c4-915c-e69dfc9688e2-istio-podinfo\") pod \"996905d6-94af-41c4-915c-e69dfc9688e2\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " Nov 02 23:43:59.704080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.703924 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-data" pod "996905d6-94af-41c4-915c-e69dfc9688e2" (UID: "996905d6-94af-41c4-915c-e69dfc9688e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:59.704080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.703955 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-socket\") pod \"996905d6-94af-41c4-915c-e69dfc9688e2\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " Nov 02 23:43:59.704080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.703990 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-istio-token\" (UniqueName: \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-istio-token\") pod \"996905d6-94af-41c4-915c-e69dfc9688e2\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " Nov 02 23:43:59.704080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.704054 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-certs\") pod \"996905d6-94af-41c4-915c-e69dfc9688e2\" (UID: \"996905d6-94af-41c4-915c-e69dfc9688e2\") " Nov 02 23:43:59.704356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.704324 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-credential-socket" pod "996905d6-94af-41c4-915c-e69dfc9688e2" (UID: "996905d6-94af-41c4-915c-e69dfc9688e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:59.704419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.704256 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996905d6-94af-41c4-915c-e69dfc9688e2-istiod-ca-cert" pod "996905d6-94af-41c4-915c-e69dfc9688e2" (UID: "996905d6-94af-41c4-915c-e69dfc9688e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:43:59.704584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.704542 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-certs" pod "996905d6-94af-41c4-915c-e69dfc9688e2" (UID: "996905d6-94af-41c4-915c-e69dfc9688e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:59.704584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.704557 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-socket" pod "996905d6-94af-41c4-915c-e69dfc9688e2" (UID: "996905d6-94af-41c4-915c-e69dfc9688e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:59.706127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.704632 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.706127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.705920 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996905d6-94af-41c4-915c-e69dfc9688e2-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.706127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.705939 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.708056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.706739 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-envoy" pod "996905d6-94af-41c4-915c-e69dfc9688e2" (UID: "996905d6-94af-41c4-915c-e69dfc9688e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:43:59.708056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.706755 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-istio-token" pod "996905d6-94af-41c4-915c-e69dfc9688e2" (UID: "996905d6-94af-41c4-915c-e69dfc9688e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:59.719678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.718863 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-kube-api-access-plvp2" pod "996905d6-94af-41c4-915c-e69dfc9688e2" (UID: "996905d6-94af-41c4-915c-e69dfc9688e2"). InnerVolumeSpecName "kube-api-access-plvp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:43:59.719678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.718945 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/996905d6-94af-41c4-915c-e69dfc9688e2-istio-podinfo" pod "996905d6-94af-41c4-915c-e69dfc9688e2" (UID: "996905d6-94af-41c4-915c-e69dfc9688e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:43:59.807535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.807428 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996905d6-94af-41c4-915c-e69dfc9688e2-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.807535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.807478 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.807535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.807495 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.807535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.807512 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.807535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.807534 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996905d6-94af-41c4-915c-e69dfc9688e2-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.807854 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.807556 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plvp2\" (UniqueName: \"kubernetes.io/projected/996905d6-94af-41c4-915c-e69dfc9688e2-kube-api-access-plvp2\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:43:59.810442 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.810415 5672 generic.go:356] "Generic (PLEG): container finished" podID="996905d6-94af-41c4-915c-e69dfc9688e2" containerID="5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e" exitCode=0 Nov 02 23:43:59.810573 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.810479 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" Nov 02 23:43:59.810573 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.810476 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" event={"ID":"996905d6-94af-41c4-915c-e69dfc9688e2","Type":"ContainerDied","Data":"5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e"} Nov 02 23:43:59.810573 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.810546 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9" event={"ID":"996905d6-94af-41c4-915c-e69dfc9688e2","Type":"ContainerDied","Data":"da59dd26874f8b287472fa7be553c0c7860b4c92858415b4b4eec7842558ee4b"} Nov 02 23:43:59.810683 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.810579 5672 scope.go:122] "RemoveContainer" containerID="5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e" Nov 02 23:43:59.821422 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.821406 5672 scope.go:122] "RemoveContainer" containerID="5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e" Nov 02 23:43:59.821737 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:43:59.821717 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e\": container with ID starting with 5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e not found: ID does not exist" containerID="5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e" Nov 02 23:43:59.821825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.821748 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e"} err="failed to get container status \"5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e\": rpc error: code = NotFound desc = could not find container \"5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e\": container with ID starting with 5332eb6de86eb5dd1c9fe2d04323accf2dccc8890a547f8b1693fa7505eb943e not found: ID does not exist" Nov 02 23:43:59.964533 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.964451 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9"] Nov 02 23:43:59.969694 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:43:59.969664 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-ts7t9"] Nov 02 23:44:00.713656 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:00.713622 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996905d6-94af-41c4-915c-e69dfc9688e2" path="/var/lib/kubelet/pods/996905d6-94af-41c4-915c-e69dfc9688e2/volumes" Nov 02 23:44:12.903410 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:12.903373 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:44:26.032308 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:44:26.032266 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:44:28.797166 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:28.797110 5672 scope.go:122] "RemoveContainer" containerID="1481614ffe55b1e108b46e96f80a002d9a18c7fbd7a78238e2936558f6791fca" Nov 02 23:44:28.807077 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:28.807056 5672 scope.go:122] "RemoveContainer" containerID="c184dba65f550b42de36864c9a97b15c4d27b42c7bbea4bb4b25432a2470442b" Nov 02 23:44:28.816208 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:28.816189 5672 scope.go:122] "RemoveContainer" containerID="3f9f8f5957488eff74c6ab5ca76a9a023b3b2ae612f96babdb55cdc7ded5f453" Nov 02 23:44:28.828583 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:28.828555 5672 scope.go:122] "RemoveContainer" containerID="1df5a82a5b7ad7e11506921e7a3bf13c7ba5828c322347ebbf2bd02b52cf6d7b" Nov 02 23:44:28.837997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:28.837972 5672 scope.go:122] "RemoveContainer" containerID="b85dbe0a26ea3442ad375622ab7774307ce071b294ac1063754fe58433bd85d2" Nov 02 23:44:32.587917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.587880 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8"] Nov 02 23:44:32.588512 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.588484 5672 cpu_manager.go:414] "containerMap: removing container" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" Nov 02 23:44:32.588512 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.588507 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" Nov 02 23:44:32.588717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.588533 5672 cpu_manager.go:414] "containerMap: removing container" podUID="996905d6-94af-41c4-915c-e69dfc9688e2" containerName="istio-proxy" Nov 02 23:44:32.588717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.588542 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996905d6-94af-41c4-915c-e69dfc9688e2" containerName="istio-proxy" Nov 02 23:44:32.588717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.588688 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="afabbfc9-a7bd-43f6-8923-83c13a54d7ac" containerName="istio-proxy" Nov 02 23:44:32.588717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.588709 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="996905d6-94af-41c4-915c-e69dfc9688e2" containerName="istio-proxy" Nov 02 23:44:32.593226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.593205 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.648216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.648182 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8"] Nov 02 23:44:32.688346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.688318 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-c25940\"/\"openshift-service-ca.crt\"" Nov 02 23:44:32.688346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.688319 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-c25940\"/\"kube-root-ca.crt\"" Nov 02 23:44:32.688544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.688377 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-c25940\"/\"default-dockercfg-l8cwh\"" Nov 02 23:44:32.688968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.688944 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-c25940\"/\"auth-test-model-c25940-kserve-self-signed-certs\"" Nov 02 23:44:32.703532 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.703510 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-tmp-dir\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.703641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.703544 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105a58d0-a50d-4ef8-a450-4778c6b97201-tls-certs\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.703641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.703567 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-home\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.703752 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.703731 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-dshm\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.703796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.703768 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-model-cache\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.703876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.703794 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbh59\" (UniqueName: \"kubernetes.io/projected/105a58d0-a50d-4ef8-a450-4778c6b97201-kube-api-access-mbh59\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.804714 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.804678 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-dshm\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.804932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.804729 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-model-cache\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.804932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.804762 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbh59\" (UniqueName: \"kubernetes.io/projected/105a58d0-a50d-4ef8-a450-4778c6b97201-kube-api-access-mbh59\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.804932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.804876 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-tmp-dir\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.804932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.804913 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105a58d0-a50d-4ef8-a450-4778c6b97201-tls-certs\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.805218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.804945 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-home\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.805218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.805205 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-model-cache\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.805421 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.805397 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-tmp-dir\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.805600 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.805574 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-home\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.807134 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.807110 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-dshm\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.807583 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.807564 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105a58d0-a50d-4ef8-a450-4778c6b97201-tls-certs\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.862873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.862794 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbh59\" (UniqueName: \"kubernetes.io/projected/105a58d0-a50d-4ef8-a450-4778c6b97201-kube-api-access-mbh59\") pod \"auth-test-model-c25940-kserve-6fdf77cd89-5cmx8\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:32.905431 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:32.905382 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:33.284101 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:33.284062 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8"] Nov 02 23:44:33.285487 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:44:33.285459 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod105a58d0_a50d_4ef8_a450_4778c6b97201.slice/crio-fb14ca17f612f35f722b041b4b9e0232536020117fa50943c024efbaf1241b30 WatchSource:0}: Error finding container fb14ca17f612f35f722b041b4b9e0232536020117fa50943c024efbaf1241b30: Status 404 returned error can't find the container with id fb14ca17f612f35f722b041b4b9e0232536020117fa50943c024efbaf1241b30 Nov 02 23:44:34.096487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:34.096451 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" event={"ID":"105a58d0-a50d-4ef8-a450-4778c6b97201","Type":"ContainerStarted","Data":"2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6"} Nov 02 23:44:34.096894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:34.096500 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" event={"ID":"105a58d0-a50d-4ef8-a450-4778c6b97201","Type":"ContainerStarted","Data":"fb14ca17f612f35f722b041b4b9e0232536020117fa50943c024efbaf1241b30"} Nov 02 23:44:34.203947 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:34.203890 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" podStartSLOduration=2.203875632 podStartE2EDuration="2.203875632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:44:34.202584315 +0000 UTC m=+3067.609896158" watchObservedRunningTime="2026-11-02 23:44:34.203875632 +0000 UTC m=+3067.611187509" Nov 02 23:44:34.905623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:34.905588 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:42.905735 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:42.905698 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:42.918210 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:42.918186 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:43.178188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:44:43.178073 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:44:56.020571 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:44:56.020536 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:45:00.166434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.166396 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq"] Nov 02 23:45:00.171134 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.171098 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c"] Nov 02 23:45:00.171301 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.171257 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" Nov 02 23:45:00.175641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.175615 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc"] Nov 02 23:45:00.175756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.175719 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" Nov 02 23:45:00.179141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.179123 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" Nov 02 23:45:00.211407 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.211378 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb"] Nov 02 23:45:00.215295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.215274 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp"] Nov 02 23:45:00.215408 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.215395 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" Nov 02 23:45:00.218970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.218950 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn"] Nov 02 23:45:00.219111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.219079 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" Nov 02 23:45:00.222884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.222864 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx"] Nov 02 23:45:00.222999 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.222979 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" Nov 02 23:45:00.226555 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.226535 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt"] Nov 02 23:45:00.226650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.226634 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" Nov 02 23:45:00.230978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.230958 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw"] Nov 02 23:45:00.231093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.231074 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" Nov 02 23:45:00.235411 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.235392 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv"] Nov 02 23:45:00.235526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.235509 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" Nov 02 23:45:00.240589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.240571 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" Nov 02 23:45:00.242572 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.242550 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq\" (UID: \"c9f89391-7aab-40c8-95d8-ae30eb9b2f50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" Nov 02 23:45:00.242715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.242693 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkjmv\" (UniqueName: \"kubernetes.io/projected/517d111c-1dd2-4e24-99ab-ac1556f891ae-kube-api-access-mkjmv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc\" (UID: \"517d111c-1dd2-4e24-99ab-ac1556f891ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" Nov 02 23:45:00.242855 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.242813 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq92h\" (UniqueName: \"kubernetes.io/projected/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-kube-api-access-tq92h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq\" (UID: \"c9f89391-7aab-40c8-95d8-ae30eb9b2f50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" Nov 02 23:45:00.242923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.242896 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vml54\" (UniqueName: \"kubernetes.io/projected/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-kube-api-access-vml54\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c\" (UID: \"e1a4d4c2-a470-4945-85a4-2ceec73fd11b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" Nov 02 23:45:00.242986 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.242959 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/517d111c-1dd2-4e24-99ab-ac1556f891ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc\" (UID: \"517d111c-1dd2-4e24-99ab-ac1556f891ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" Nov 02 23:45:00.243055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.243035 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c\" (UID: \"e1a4d4c2-a470-4945-85a4-2ceec73fd11b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" Nov 02 23:45:00.255375 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.255356 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:45:00.255534 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.255382 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:45:00.255606 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.255570 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:45:00.258272 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.258251 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq"] Nov 02 23:45:00.262447 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.262313 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c"] Nov 02 23:45:00.266158 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.266125 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb"] Nov 02 23:45:00.307492 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.307448 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp"] Nov 02 23:45:00.318263 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.318231 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx"] Nov 02 23:45:00.320114 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.320088 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt"] Nov 02 23:45:00.344056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344014 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjmv\" (UniqueName: \"kubernetes.io/projected/517d111c-1dd2-4e24-99ab-ac1556f891ae-kube-api-access-mkjmv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc\" (UID: \"517d111c-1dd2-4e24-99ab-ac1556f891ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" Nov 02 23:45:00.344056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344056 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90894c0e-93e0-4e47-b550-80b4b66f091e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw\" (UID: \"90894c0e-93e0-4e47-b550-80b4b66f091e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" Nov 02 23:45:00.344389 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344093 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slrtb\" (UniqueName: \"kubernetes.io/projected/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-kube-api-access-slrtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp\" (UID: \"558a2c75-72b2-41cf-b3e4-9d2dc53437f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" Nov 02 23:45:00.344389 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344123 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd99463-1bb3-4b55-a805-49dcae325120-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb\" (UID: \"bdd99463-1bb3-4b55-a805-49dcae325120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" Nov 02 23:45:00.344389 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344287 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g89zp\" (UniqueName: \"kubernetes.io/projected/6d12bc18-bd4d-495b-8222-f9682c97db74-kube-api-access-g89zp\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv\" (UID: \"6d12bc18-bd4d-495b-8222-f9682c97db74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" Nov 02 23:45:00.344389 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344336 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq92h\" (UniqueName: \"kubernetes.io/projected/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-kube-api-access-tq92h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq\" (UID: \"c9f89391-7aab-40c8-95d8-ae30eb9b2f50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" Nov 02 23:45:00.344631 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344383 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99vz\" (UniqueName: \"kubernetes.io/projected/90894c0e-93e0-4e47-b550-80b4b66f091e-kube-api-access-d99vz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw\" (UID: \"90894c0e-93e0-4e47-b550-80b4b66f091e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" Nov 02 23:45:00.344631 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344430 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d12bc18-bd4d-495b-8222-f9682c97db74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv\" (UID: \"6d12bc18-bd4d-495b-8222-f9682c97db74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" Nov 02 23:45:00.344631 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344509 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vml54\" (UniqueName: \"kubernetes.io/projected/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-kube-api-access-vml54\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c\" (UID: \"e1a4d4c2-a470-4945-85a4-2ceec73fd11b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" Nov 02 23:45:00.344631 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344576 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/517d111c-1dd2-4e24-99ab-ac1556f891ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc\" (UID: \"517d111c-1dd2-4e24-99ab-ac1556f891ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" Nov 02 23:45:00.344631 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344616 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef0b79-c23e-45ff-8936-92f4f0ea3708-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894385-wc8cx\" (UID: \"7cef0b79-c23e-45ff-8936-92f4f0ea3708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" Nov 02 23:45:00.344923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344651 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp\" (UID: \"558a2c75-72b2-41cf-b3e4-9d2dc53437f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" Nov 02 23:45:00.344923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344732 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c\" (UID: \"e1a4d4c2-a470-4945-85a4-2ceec73fd11b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" Nov 02 23:45:00.344923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344806 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq\" (UID: \"c9f89391-7aab-40c8-95d8-ae30eb9b2f50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" Nov 02 23:45:00.344923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344849 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfdxg\" (UniqueName: \"kubernetes.io/projected/bdd99463-1bb3-4b55-a805-49dcae325120-kube-api-access-pfdxg\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb\" (UID: \"bdd99463-1bb3-4b55-a805-49dcae325120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" Nov 02 23:45:00.344923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344893 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn\" (UID: \"f7a89c90-63d6-44ab-a0a5-159fa8096ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" Nov 02 23:45:00.345221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344940 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs99g\" (UniqueName: \"kubernetes.io/projected/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-kube-api-access-qs99g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn\" (UID: \"f7a89c90-63d6-44ab-a0a5-159fa8096ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" Nov 02 23:45:00.345221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.344975 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kcjs\" (UniqueName: \"kubernetes.io/projected/7cef0b79-c23e-45ff-8936-92f4f0ea3708-kube-api-access-2kcjs\") pod \"maas-api-key-cleanup-29894385-wc8cx\" (UID: \"7cef0b79-c23e-45ff-8936-92f4f0ea3708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" Nov 02 23:45:00.345221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.345000 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ed4836-7860-47fc-be7e-df5a7735b640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt\" (UID: \"d4ed4836-7860-47fc-be7e-df5a7735b640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" Nov 02 23:45:00.345221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.345041 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbgjk\" (UniqueName: \"kubernetes.io/projected/d4ed4836-7860-47fc-be7e-df5a7735b640-kube-api-access-gbgjk\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt\" (UID: \"d4ed4836-7860-47fc-be7e-df5a7735b640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" Nov 02 23:45:00.345453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.345416 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c\" (UID: \"e1a4d4c2-a470-4945-85a4-2ceec73fd11b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" Nov 02 23:45:00.345453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.345416 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/517d111c-1dd2-4e24-99ab-ac1556f891ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc\" (UID: \"517d111c-1dd2-4e24-99ab-ac1556f891ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" Nov 02 23:45:00.345853 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.345828 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq\" (UID: \"c9f89391-7aab-40c8-95d8-ae30eb9b2f50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" Nov 02 23:45:00.355362 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.355332 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw"] Nov 02 23:45:00.365662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.364675 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv"] Nov 02 23:45:00.378635 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.375639 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc"] Nov 02 23:45:00.379044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.379024 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn"] Nov 02 23:45:00.397525 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.397493 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vml54\" (UniqueName: \"kubernetes.io/projected/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-kube-api-access-vml54\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c\" (UID: \"e1a4d4c2-a470-4945-85a4-2ceec73fd11b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" Nov 02 23:45:00.397953 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.397936 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq92h\" (UniqueName: \"kubernetes.io/projected/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-kube-api-access-tq92h\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq\" (UID: \"c9f89391-7aab-40c8-95d8-ae30eb9b2f50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" Nov 02 23:45:00.397999 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.397984 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjmv\" (UniqueName: \"kubernetes.io/projected/517d111c-1dd2-4e24-99ab-ac1556f891ae-kube-api-access-mkjmv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc\" (UID: \"517d111c-1dd2-4e24-99ab-ac1556f891ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.447448 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbgjk\" (UniqueName: \"kubernetes.io/projected/d4ed4836-7860-47fc-be7e-df5a7735b640-kube-api-access-gbgjk\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt\" (UID: \"d4ed4836-7860-47fc-be7e-df5a7735b640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.447513 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90894c0e-93e0-4e47-b550-80b4b66f091e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw\" (UID: \"90894c0e-93e0-4e47-b550-80b4b66f091e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.447570 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrtb\" (UniqueName: \"kubernetes.io/projected/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-kube-api-access-slrtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp\" (UID: \"558a2c75-72b2-41cf-b3e4-9d2dc53437f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.447612 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd99463-1bb3-4b55-a805-49dcae325120-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb\" (UID: \"bdd99463-1bb3-4b55-a805-49dcae325120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.447664 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g89zp\" (UniqueName: \"kubernetes.io/projected/6d12bc18-bd4d-495b-8222-f9682c97db74-kube-api-access-g89zp\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv\" (UID: \"6d12bc18-bd4d-495b-8222-f9682c97db74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.447709 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99vz\" (UniqueName: \"kubernetes.io/projected/90894c0e-93e0-4e47-b550-80b4b66f091e-kube-api-access-d99vz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw\" (UID: \"90894c0e-93e0-4e47-b550-80b4b66f091e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.447746 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d12bc18-bd4d-495b-8222-f9682c97db74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv\" (UID: \"6d12bc18-bd4d-495b-8222-f9682c97db74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.447830 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef0b79-c23e-45ff-8936-92f4f0ea3708-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894385-wc8cx\" (UID: \"7cef0b79-c23e-45ff-8936-92f4f0ea3708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.447869 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp\" (UID: \"558a2c75-72b2-41cf-b3e4-9d2dc53437f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.447957 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfdxg\" (UniqueName: \"kubernetes.io/projected/bdd99463-1bb3-4b55-a805-49dcae325120-kube-api-access-pfdxg\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb\" (UID: \"bdd99463-1bb3-4b55-a805-49dcae325120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.448005 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn\" (UID: \"f7a89c90-63d6-44ab-a0a5-159fa8096ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.448058 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs99g\" (UniqueName: \"kubernetes.io/projected/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-kube-api-access-qs99g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn\" (UID: \"f7a89c90-63d6-44ab-a0a5-159fa8096ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.448089 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kcjs\" (UniqueName: \"kubernetes.io/projected/7cef0b79-c23e-45ff-8936-92f4f0ea3708-kube-api-access-2kcjs\") pod \"maas-api-key-cleanup-29894385-wc8cx\" (UID: \"7cef0b79-c23e-45ff-8936-92f4f0ea3708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.448120 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ed4836-7860-47fc-be7e-df5a7735b640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt\" (UID: \"d4ed4836-7860-47fc-be7e-df5a7735b640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.448963 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ed4836-7860-47fc-be7e-df5a7735b640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt\" (UID: \"d4ed4836-7860-47fc-be7e-df5a7735b640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" Nov 02 23:45:00.449736 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.449061 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d12bc18-bd4d-495b-8222-f9682c97db74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv\" (UID: \"6d12bc18-bd4d-495b-8222-f9682c97db74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" Nov 02 23:45:00.458917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.449810 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef0b79-c23e-45ff-8936-92f4f0ea3708-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894385-wc8cx\" (UID: \"7cef0b79-c23e-45ff-8936-92f4f0ea3708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" Nov 02 23:45:00.458917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.450345 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90894c0e-93e0-4e47-b550-80b4b66f091e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw\" (UID: \"90894c0e-93e0-4e47-b550-80b4b66f091e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" Nov 02 23:45:00.459035 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.458987 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn\" (UID: \"f7a89c90-63d6-44ab-a0a5-159fa8096ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" Nov 02 23:45:00.459089 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.459044 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd99463-1bb3-4b55-a805-49dcae325120-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb\" (UID: \"bdd99463-1bb3-4b55-a805-49dcae325120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" Nov 02 23:45:00.459162 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.459124 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp\" (UID: \"558a2c75-72b2-41cf-b3e4-9d2dc53437f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" Nov 02 23:45:00.487014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.486987 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" Nov 02 23:45:00.495208 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.495183 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" Nov 02 23:45:00.501487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.501270 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" Nov 02 23:45:00.502672 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.502646 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbgjk\" (UniqueName: \"kubernetes.io/projected/d4ed4836-7860-47fc-be7e-df5a7735b640-kube-api-access-gbgjk\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt\" (UID: \"d4ed4836-7860-47fc-be7e-df5a7735b640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" Nov 02 23:45:00.514276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.514244 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrtb\" (UniqueName: \"kubernetes.io/projected/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-kube-api-access-slrtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp\" (UID: \"558a2c75-72b2-41cf-b3e4-9d2dc53437f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" Nov 02 23:45:00.514408 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.514254 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99vz\" (UniqueName: \"kubernetes.io/projected/90894c0e-93e0-4e47-b550-80b4b66f091e-kube-api-access-d99vz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw\" (UID: \"90894c0e-93e0-4e47-b550-80b4b66f091e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" Nov 02 23:45:00.516296 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.516273 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfdxg\" (UniqueName: \"kubernetes.io/projected/bdd99463-1bb3-4b55-a805-49dcae325120-kube-api-access-pfdxg\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb\" (UID: \"bdd99463-1bb3-4b55-a805-49dcae325120\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" Nov 02 23:45:00.522219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.522120 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs99g\" (UniqueName: \"kubernetes.io/projected/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-kube-api-access-qs99g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn\" (UID: \"f7a89c90-63d6-44ab-a0a5-159fa8096ca5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" Nov 02 23:45:00.522219 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.522126 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kcjs\" (UniqueName: \"kubernetes.io/projected/7cef0b79-c23e-45ff-8936-92f4f0ea3708-kube-api-access-2kcjs\") pod \"maas-api-key-cleanup-29894385-wc8cx\" (UID: \"7cef0b79-c23e-45ff-8936-92f4f0ea3708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" Nov 02 23:45:00.524541 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.524514 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g89zp\" (UniqueName: \"kubernetes.io/projected/6d12bc18-bd4d-495b-8222-f9682c97db74-kube-api-access-g89zp\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv\" (UID: \"6d12bc18-bd4d-495b-8222-f9682c97db74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" Nov 02 23:45:00.534340 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.534225 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" Nov 02 23:45:00.541773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.540311 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" Nov 02 23:45:00.556531 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.556500 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" Nov 02 23:45:00.562961 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.562914 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" Nov 02 23:45:00.581681 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.579882 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" Nov 02 23:45:00.602242 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.601969 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" Nov 02 23:45:00.603434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.603299 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" Nov 02 23:45:00.900688 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:00.899413 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq"] Nov 02 23:45:01.014677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.014646 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb"] Nov 02 23:45:01.014893 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:45:01.014691 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd99463_1bb3_4b55_a805_49dcae325120.slice/crio-e1331a436c080dabf4de1bdce10a9c8c4b16a8afefd5f1f349dcb6ab9d8a971d WatchSource:0}: Error finding container e1331a436c080dabf4de1bdce10a9c8c4b16a8afefd5f1f349dcb6ab9d8a971d: Status 404 returned error can't find the container with id e1331a436c080dabf4de1bdce10a9c8c4b16a8afefd5f1f349dcb6ab9d8a971d Nov 02 23:45:01.079922 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.079895 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp"] Nov 02 23:45:01.080791 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:45:01.080766 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558a2c75_72b2_41cf_b3e4_9d2dc53437f3.slice/crio-4abfdd41c4a04d1c0a9817509b81cf2ad7517d483de155461629a08b6f1a0108 WatchSource:0}: Error finding container 4abfdd41c4a04d1c0a9817509b81cf2ad7517d483de155461629a08b6f1a0108: Status 404 returned error can't find the container with id 4abfdd41c4a04d1c0a9817509b81cf2ad7517d483de155461629a08b6f1a0108 Nov 02 23:45:01.093977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.093956 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx"] Nov 02 23:45:01.095548 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:45:01.095507 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cef0b79_c23e_45ff_8936_92f4f0ea3708.slice/crio-cd7437933f420109b8466984f0da3d76aa1974cafd3eab4c5decef51381a00e1 WatchSource:0}: Error finding container cd7437933f420109b8466984f0da3d76aa1974cafd3eab4c5decef51381a00e1: Status 404 returned error can't find the container with id cd7437933f420109b8466984f0da3d76aa1974cafd3eab4c5decef51381a00e1 Nov 02 23:45:01.116768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.116741 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c"] Nov 02 23:45:01.117760 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:45:01.117734 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a4d4c2_a470_4945_85a4_2ceec73fd11b.slice/crio-67fcaf53cd90acf7ab8ba2b1d0ca712f936b735621f1e01db3fc35ce7d987737 WatchSource:0}: Error finding container 67fcaf53cd90acf7ab8ba2b1d0ca712f936b735621f1e01db3fc35ce7d987737: Status 404 returned error can't find the container with id 67fcaf53cd90acf7ab8ba2b1d0ca712f936b735621f1e01db3fc35ce7d987737 Nov 02 23:45:01.145609 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.145420 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc"] Nov 02 23:45:01.165232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.165200 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw"] Nov 02 23:45:01.185249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.185225 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv"] Nov 02 23:45:01.229373 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:45:01.229340 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517d111c_1dd2_4e24_99ab_ac1556f891ae.slice/crio-d8282dc7638cfdb8e5fde5964369d8c510ad04bfc16aacce2a36b11f48daab4b WatchSource:0}: Error finding container d8282dc7638cfdb8e5fde5964369d8c510ad04bfc16aacce2a36b11f48daab4b: Status 404 returned error can't find the container with id d8282dc7638cfdb8e5fde5964369d8c510ad04bfc16aacce2a36b11f48daab4b Nov 02 23:45:01.230315 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:45:01.230289 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90894c0e_93e0_4e47_b550_80b4b66f091e.slice/crio-9a11ee4eb2cde0999c86c6e34eebfa4bcf3c0c37a03b8d7ee1214319624d3fd9 WatchSource:0}: Error finding container 9a11ee4eb2cde0999c86c6e34eebfa4bcf3c0c37a03b8d7ee1214319624d3fd9: Status 404 returned error can't find the container with id 9a11ee4eb2cde0999c86c6e34eebfa4bcf3c0c37a03b8d7ee1214319624d3fd9 Nov 02 23:45:01.283753 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.283674 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" event={"ID":"6d12bc18-bd4d-495b-8222-f9682c97db74","Type":"ContainerStarted","Data":"854167b5f751ea316f53f92dc8739b736aed32ea580d46d2c480bd556e4d6f04"} Nov 02 23:45:01.284932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.284909 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" event={"ID":"e1a4d4c2-a470-4945-85a4-2ceec73fd11b","Type":"ContainerStarted","Data":"67fcaf53cd90acf7ab8ba2b1d0ca712f936b735621f1e01db3fc35ce7d987737"} Nov 02 23:45:01.286098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.286069 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" event={"ID":"bdd99463-1bb3-4b55-a805-49dcae325120","Type":"ContainerStarted","Data":"e1331a436c080dabf4de1bdce10a9c8c4b16a8afefd5f1f349dcb6ab9d8a971d"} Nov 02 23:45:01.287290 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.287265 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" event={"ID":"558a2c75-72b2-41cf-b3e4-9d2dc53437f3","Type":"ContainerStarted","Data":"4abfdd41c4a04d1c0a9817509b81cf2ad7517d483de155461629a08b6f1a0108"} Nov 02 23:45:01.288423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.288404 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" event={"ID":"7cef0b79-c23e-45ff-8936-92f4f0ea3708","Type":"ContainerStarted","Data":"cd7437933f420109b8466984f0da3d76aa1974cafd3eab4c5decef51381a00e1"} Nov 02 23:45:01.289510 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.289493 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" event={"ID":"c9f89391-7aab-40c8-95d8-ae30eb9b2f50","Type":"ContainerStarted","Data":"da3a1bb08cedcf5db58c077ff9a233e2f77afb3d37e3c11e25d881b092501a12"} Nov 02 23:45:01.290601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.290579 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" event={"ID":"517d111c-1dd2-4e24-99ab-ac1556f891ae","Type":"ContainerStarted","Data":"d8282dc7638cfdb8e5fde5964369d8c510ad04bfc16aacce2a36b11f48daab4b"} Nov 02 23:45:01.291670 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.291648 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" event={"ID":"90894c0e-93e0-4e47-b550-80b4b66f091e","Type":"ContainerStarted","Data":"9a11ee4eb2cde0999c86c6e34eebfa4bcf3c0c37a03b8d7ee1214319624d3fd9"} Nov 02 23:45:01.297273 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.297254 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt"] Nov 02 23:45:01.300760 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:01.300740 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn"] Nov 02 23:45:01.332704 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:45:01.332594 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ed4836_7860_47fc_be7e_df5a7735b640.slice/crio-05a75e2da61d530f4b637c492d301ad954f507a097af1249a5f24c1fb6f8e113 WatchSource:0}: Error finding container 05a75e2da61d530f4b637c492d301ad954f507a097af1249a5f24c1fb6f8e113: Status 404 returned error can't find the container with id 05a75e2da61d530f4b637c492d301ad954f507a097af1249a5f24c1fb6f8e113 Nov 02 23:45:01.333829 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:45:01.333805 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a89c90_63d6_44ab_a0a5_159fa8096ca5.slice/crio-e4d542c3ca67041f8c60fd0db4fc38b25916318c30e1922138caf691826618bc WatchSource:0}: Error finding container e4d542c3ca67041f8c60fd0db4fc38b25916318c30e1922138caf691826618bc: Status 404 returned error can't find the container with id e4d542c3ca67041f8c60fd0db4fc38b25916318c30e1922138caf691826618bc Nov 02 23:45:02.309216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.309175 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" event={"ID":"e1a4d4c2-a470-4945-85a4-2ceec73fd11b","Type":"ContainerStarted","Data":"24dba64a7dc394f21ed1d1a3455a481b375f3391744e96210723d8f9de0f7aa8"} Nov 02 23:45:02.310911 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.310884 5672 generic.go:356] "Generic (PLEG): container finished" podID="bdd99463-1bb3-4b55-a805-49dcae325120" containerID="6e4ee777e1ed427c8d1fe7b6fbf57ee917f2af077187e21615851cb1b0960ccb" exitCode=60 Nov 02 23:45:02.311042 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.310968 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" event={"ID":"bdd99463-1bb3-4b55-a805-49dcae325120","Type":"ContainerDied","Data":"6e4ee777e1ed427c8d1fe7b6fbf57ee917f2af077187e21615851cb1b0960ccb"} Nov 02 23:45:02.311419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.311173 5672 scope.go:122] "RemoveContainer" containerID="6e4ee777e1ed427c8d1fe7b6fbf57ee917f2af077187e21615851cb1b0960ccb" Nov 02 23:45:02.312579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.312550 5672 generic.go:356] "Generic (PLEG): container finished" podID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerID="160373e662a5255ee74af352b9764a1e2227e5241566b773b7c4f651293faefb" exitCode=60 Nov 02 23:45:02.312579 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.312571 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" event={"ID":"558a2c75-72b2-41cf-b3e4-9d2dc53437f3","Type":"ContainerDied","Data":"160373e662a5255ee74af352b9764a1e2227e5241566b773b7c4f651293faefb"} Nov 02 23:45:02.312833 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.312806 5672 scope.go:122] "RemoveContainer" containerID="160373e662a5255ee74af352b9764a1e2227e5241566b773b7c4f651293faefb" Nov 02 23:45:02.314258 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.314182 5672 generic.go:356] "Generic (PLEG): container finished" podID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerID="f6352d65eeaac2095ccecfe295a97c73241d108b6d82cb9796a4aa390c828f76" exitCode=60 Nov 02 23:45:02.314258 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.314252 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" event={"ID":"7cef0b79-c23e-45ff-8936-92f4f0ea3708","Type":"ContainerDied","Data":"f6352d65eeaac2095ccecfe295a97c73241d108b6d82cb9796a4aa390c828f76"} Nov 02 23:45:02.314499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.314457 5672 scope.go:122] "RemoveContainer" containerID="f6352d65eeaac2095ccecfe295a97c73241d108b6d82cb9796a4aa390c828f76" Nov 02 23:45:02.316052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.315946 5672 generic.go:356] "Generic (PLEG): container finished" podID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerID="ac651872be2ad1d4266ab700ca70cfaf496a30c9f560e6bf5472e792b1dde5e5" exitCode=60 Nov 02 23:45:02.316052 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.316019 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" event={"ID":"c9f89391-7aab-40c8-95d8-ae30eb9b2f50","Type":"ContainerDied","Data":"ac651872be2ad1d4266ab700ca70cfaf496a30c9f560e6bf5472e792b1dde5e5"} Nov 02 23:45:02.316220 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.316186 5672 scope.go:122] "RemoveContainer" containerID="ac651872be2ad1d4266ab700ca70cfaf496a30c9f560e6bf5472e792b1dde5e5" Nov 02 23:45:02.317519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.317498 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" event={"ID":"517d111c-1dd2-4e24-99ab-ac1556f891ae","Type":"ContainerStarted","Data":"a6b939b33b966ffb613da34c5d886e861569791415ad050a9b51f4fb46b0463c"} Nov 02 23:45:02.319102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.319077 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" event={"ID":"f7a89c90-63d6-44ab-a0a5-159fa8096ca5","Type":"ContainerStarted","Data":"d7b0432240d6f1181e43cd0efc86a6be21a989f144634345bca0123612963251"} Nov 02 23:45:02.319226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.319113 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" event={"ID":"f7a89c90-63d6-44ab-a0a5-159fa8096ca5","Type":"ContainerStarted","Data":"e4d542c3ca67041f8c60fd0db4fc38b25916318c30e1922138caf691826618bc"} Nov 02 23:45:02.320801 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.320781 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" event={"ID":"90894c0e-93e0-4e47-b550-80b4b66f091e","Type":"ContainerStarted","Data":"13a72e8ebf06eb65eefaf917287e8caf37f23161b9b256663b85c3708a32c878"} Nov 02 23:45:02.322520 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.322466 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" event={"ID":"d4ed4836-7860-47fc-be7e-df5a7735b640","Type":"ContainerStarted","Data":"75c49051dbd56b36641d3a8a04bdd500360116f48d87ef1cd1b9017bce161b5b"} Nov 02 23:45:02.322604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.322547 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" event={"ID":"d4ed4836-7860-47fc-be7e-df5a7735b640","Type":"ContainerStarted","Data":"05a75e2da61d530f4b637c492d301ad954f507a097af1249a5f24c1fb6f8e113"} Nov 02 23:45:02.324132 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.324110 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" event={"ID":"6d12bc18-bd4d-495b-8222-f9682c97db74","Type":"ContainerStarted","Data":"c8e6cf8c559e5bbcd2d37b7dcc9e74f54cac2979ce9757a1be62c2f5dd8f4a55"} Nov 02 23:45:02.420030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.419983 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" podStartSLOduration=2.419969684 podStartE2EDuration="2.419969684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.417288528 +0000 UTC m=+3095.824600361" watchObservedRunningTime="2026-11-02 23:45:02.419969684 +0000 UTC m=+3095.827281508" Nov 02 23:45:02.530269 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.529229 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" podStartSLOduration=2.529209112 podStartE2EDuration="2.529209112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.528320686 +0000 UTC m=+3095.935632518" watchObservedRunningTime="2026-11-02 23:45:02.529209112 +0000 UTC m=+3095.936520946" Nov 02 23:45:02.640091 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.640036 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" podStartSLOduration=2.639978863 podStartE2EDuration="2.639978863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.636985355 +0000 UTC m=+3096.044297190" watchObservedRunningTime="2026-11-02 23:45:02.639978863 +0000 UTC m=+3096.047290693" Nov 02 23:45:02.851172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:02.849644 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" podStartSLOduration=2.849627132 podStartE2EDuration="2.849627132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:02.848882748 +0000 UTC m=+3096.256194579" watchObservedRunningTime="2026-11-02 23:45:02.849627132 +0000 UTC m=+3096.256938963" Nov 02 23:45:03.062997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.062937 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" podStartSLOduration=3.062918958 podStartE2EDuration="3.062918958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:03.05809934 +0000 UTC m=+3096.465411172" watchObservedRunningTime="2026-11-02 23:45:03.062918958 +0000 UTC m=+3096.470230789" Nov 02 23:45:03.271903 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.271852 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" podStartSLOduration=3.271834141 podStartE2EDuration="3.271834141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:03.270620057 +0000 UTC m=+3096.677931891" watchObservedRunningTime="2026-11-02 23:45:03.271834141 +0000 UTC m=+3096.679145972" Nov 02 23:45:03.335996 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.335953 5672 generic.go:356] "Generic (PLEG): container finished" podID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerID="24dba64a7dc394f21ed1d1a3455a481b375f3391744e96210723d8f9de0f7aa8" exitCode=60 Nov 02 23:45:03.336500 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.336044 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" event={"ID":"e1a4d4c2-a470-4945-85a4-2ceec73fd11b","Type":"ContainerDied","Data":"24dba64a7dc394f21ed1d1a3455a481b375f3391744e96210723d8f9de0f7aa8"} Nov 02 23:45:03.336500 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.336372 5672 scope.go:122] "RemoveContainer" containerID="24dba64a7dc394f21ed1d1a3455a481b375f3391744e96210723d8f9de0f7aa8" Nov 02 23:45:03.338564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.338541 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" event={"ID":"bdd99463-1bb3-4b55-a805-49dcae325120","Type":"ContainerStarted","Data":"214db2607ff806d16d5662a0c7cb18fdac70bb778dc7704474040db4cfcca948"} Nov 02 23:45:03.340815 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.340790 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" event={"ID":"558a2c75-72b2-41cf-b3e4-9d2dc53437f3","Type":"ContainerStarted","Data":"fd2b95f2b477ea3cf2e85905fcec1a9ef21d8986504fed2c25e21c2265353a2c"} Nov 02 23:45:03.343580 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.343399 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" event={"ID":"7cef0b79-c23e-45ff-8936-92f4f0ea3708","Type":"ContainerStarted","Data":"e29b22a289baf4674fcdbd41827adec2beed830705a75be23ad06f9ade6c025c"} Nov 02 23:45:03.345551 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.345527 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" event={"ID":"c9f89391-7aab-40c8-95d8-ae30eb9b2f50","Type":"ContainerStarted","Data":"50106a1a4a216da55ff084c34af8a15e27777f01c515b2f8ae7f6663490fc2d5"} Nov 02 23:45:03.347345 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.347310 5672 generic.go:356] "Generic (PLEG): container finished" podID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerID="a6b939b33b966ffb613da34c5d886e861569791415ad050a9b51f4fb46b0463c" exitCode=60 Nov 02 23:45:03.347449 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.347362 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" event={"ID":"517d111c-1dd2-4e24-99ab-ac1556f891ae","Type":"ContainerDied","Data":"a6b939b33b966ffb613da34c5d886e861569791415ad050a9b51f4fb46b0463c"} Nov 02 23:45:03.347566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.347550 5672 scope.go:122] "RemoveContainer" containerID="a6b939b33b966ffb613da34c5d886e861569791415ad050a9b51f4fb46b0463c" Nov 02 23:45:03.349141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.349119 5672 generic.go:356] "Generic (PLEG): container finished" podID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerID="d7b0432240d6f1181e43cd0efc86a6be21a989f144634345bca0123612963251" exitCode=60 Nov 02 23:45:03.349261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.349244 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" event={"ID":"f7a89c90-63d6-44ab-a0a5-159fa8096ca5","Type":"ContainerDied","Data":"d7b0432240d6f1181e43cd0efc86a6be21a989f144634345bca0123612963251"} Nov 02 23:45:03.349431 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.349410 5672 scope.go:122] "RemoveContainer" containerID="d7b0432240d6f1181e43cd0efc86a6be21a989f144634345bca0123612963251" Nov 02 23:45:03.352124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.352097 5672 generic.go:356] "Generic (PLEG): container finished" podID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerID="13a72e8ebf06eb65eefaf917287e8caf37f23161b9b256663b85c3708a32c878" exitCode=60 Nov 02 23:45:03.352246 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.352216 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" event={"ID":"90894c0e-93e0-4e47-b550-80b4b66f091e","Type":"ContainerDied","Data":"13a72e8ebf06eb65eefaf917287e8caf37f23161b9b256663b85c3708a32c878"} Nov 02 23:45:03.352419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.352400 5672 scope.go:122] "RemoveContainer" containerID="13a72e8ebf06eb65eefaf917287e8caf37f23161b9b256663b85c3708a32c878" Nov 02 23:45:03.354446 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.354426 5672 generic.go:356] "Generic (PLEG): container finished" podID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerID="75c49051dbd56b36641d3a8a04bdd500360116f48d87ef1cd1b9017bce161b5b" exitCode=60 Nov 02 23:45:03.354549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.354507 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" event={"ID":"d4ed4836-7860-47fc-be7e-df5a7735b640","Type":"ContainerDied","Data":"75c49051dbd56b36641d3a8a04bdd500360116f48d87ef1cd1b9017bce161b5b"} Nov 02 23:45:03.354692 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.354677 5672 scope.go:122] "RemoveContainer" containerID="75c49051dbd56b36641d3a8a04bdd500360116f48d87ef1cd1b9017bce161b5b" Nov 02 23:45:03.356062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.356041 5672 generic.go:356] "Generic (PLEG): container finished" podID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerID="c8e6cf8c559e5bbcd2d37b7dcc9e74f54cac2979ce9757a1be62c2f5dd8f4a55" exitCode=60 Nov 02 23:45:03.356171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.356089 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" event={"ID":"6d12bc18-bd4d-495b-8222-f9682c97db74","Type":"ContainerDied","Data":"c8e6cf8c559e5bbcd2d37b7dcc9e74f54cac2979ce9757a1be62c2f5dd8f4a55"} Nov 02 23:45:03.356360 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.356342 5672 scope.go:122] "RemoveContainer" containerID="c8e6cf8c559e5bbcd2d37b7dcc9e74f54cac2979ce9757a1be62c2f5dd8f4a55" Nov 02 23:45:03.702454 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.702402 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" podStartSLOduration=3.702388744 podStartE2EDuration="3.702388744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:03.699199884 +0000 UTC m=+3097.106511716" watchObservedRunningTime="2026-11-02 23:45:03.702388744 +0000 UTC m=+3097.109700594" Nov 02 23:45:03.914712 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:03.914657 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" podStartSLOduration=3.914644167 podStartE2EDuration="3.914644167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:03.913253581 +0000 UTC m=+3097.320565410" watchObservedRunningTime="2026-11-02 23:45:03.914644167 +0000 UTC m=+3097.321956039" Nov 02 23:45:04.233371 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.233210 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" podStartSLOduration=4.2331927050000004 podStartE2EDuration="4.233192705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:04.229483861 +0000 UTC m=+3097.636795691" watchObservedRunningTime="2026-11-02 23:45:04.233192705 +0000 UTC m=+3097.640504537" Nov 02 23:45:04.365761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.365718 5672 generic.go:356] "Generic (PLEG): container finished" podID="bdd99463-1bb3-4b55-a805-49dcae325120" containerID="214db2607ff806d16d5662a0c7cb18fdac70bb778dc7704474040db4cfcca948" exitCode=60 Nov 02 23:45:04.366264 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.365803 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" event={"ID":"bdd99463-1bb3-4b55-a805-49dcae325120","Type":"ContainerDied","Data":"214db2607ff806d16d5662a0c7cb18fdac70bb778dc7704474040db4cfcca948"} Nov 02 23:45:04.366264 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.365904 5672 scope.go:122] "RemoveContainer" containerID="6e4ee777e1ed427c8d1fe7b6fbf57ee917f2af077187e21615851cb1b0960ccb" Nov 02 23:45:04.366264 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.366077 5672 scope.go:122] "RemoveContainer" containerID="214db2607ff806d16d5662a0c7cb18fdac70bb778dc7704474040db4cfcca948" Nov 02 23:45:04.366474 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:04.366449 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb_odh-ai-gateway-infra(bdd99463-1bb3-4b55-a805-49dcae325120)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" Nov 02 23:45:04.368162 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.368128 5672 generic.go:356] "Generic (PLEG): container finished" podID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerID="fd2b95f2b477ea3cf2e85905fcec1a9ef21d8986504fed2c25e21c2265353a2c" exitCode=60 Nov 02 23:45:04.368278 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.368182 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" event={"ID":"558a2c75-72b2-41cf-b3e4-9d2dc53437f3","Type":"ContainerDied","Data":"fd2b95f2b477ea3cf2e85905fcec1a9ef21d8986504fed2c25e21c2265353a2c"} Nov 02 23:45:04.368453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.368438 5672 scope.go:122] "RemoveContainer" containerID="fd2b95f2b477ea3cf2e85905fcec1a9ef21d8986504fed2c25e21c2265353a2c" Nov 02 23:45:04.368719 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:04.368695 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp_odh-ai-gateway-infra(558a2c75-72b2-41cf-b3e4-9d2dc53437f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" Nov 02 23:45:04.370417 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.370397 5672 generic.go:356] "Generic (PLEG): container finished" podID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerID="e29b22a289baf4674fcdbd41827adec2beed830705a75be23ad06f9ade6c025c" exitCode=60 Nov 02 23:45:04.370523 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.370466 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" event={"ID":"7cef0b79-c23e-45ff-8936-92f4f0ea3708","Type":"ContainerDied","Data":"e29b22a289baf4674fcdbd41827adec2beed830705a75be23ad06f9ade6c025c"} Nov 02 23:45:04.370735 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.370719 5672 scope.go:122] "RemoveContainer" containerID="e29b22a289baf4674fcdbd41827adec2beed830705a75be23ad06f9ade6c025c" Nov 02 23:45:04.371070 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:04.370991 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894385-wc8cx_odh-ai-gateway-infra(7cef0b79-c23e-45ff-8936-92f4f0ea3708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" Nov 02 23:45:04.372824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.372790 5672 generic.go:356] "Generic (PLEG): container finished" podID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerID="50106a1a4a216da55ff084c34af8a15e27777f01c515b2f8ae7f6663490fc2d5" exitCode=60 Nov 02 23:45:04.372949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.372869 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" event={"ID":"c9f89391-7aab-40c8-95d8-ae30eb9b2f50","Type":"ContainerDied","Data":"50106a1a4a216da55ff084c34af8a15e27777f01c515b2f8ae7f6663490fc2d5"} Nov 02 23:45:04.373055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.373039 5672 scope.go:122] "RemoveContainer" containerID="50106a1a4a216da55ff084c34af8a15e27777f01c515b2f8ae7f6663490fc2d5" Nov 02 23:45:04.373319 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:04.373297 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq_odh-ai-gateway-infra(c9f89391-7aab-40c8-95d8-ae30eb9b2f50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" Nov 02 23:45:04.375346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.374950 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" event={"ID":"517d111c-1dd2-4e24-99ab-ac1556f891ae","Type":"ContainerStarted","Data":"53c5f0d1a41a298fc2f0aa3013f67ebe884846e860c8ac817780f857cfbf8061"} Nov 02 23:45:04.376987 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.376964 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" event={"ID":"f7a89c90-63d6-44ab-a0a5-159fa8096ca5","Type":"ContainerStarted","Data":"d826fec642ade81feea743a31799ac2d4fd185293d631a8568587b5c1b79588b"} Nov 02 23:45:04.378864 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.378842 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" event={"ID":"90894c0e-93e0-4e47-b550-80b4b66f091e","Type":"ContainerStarted","Data":"ffb4c45896560f9e31aadf10489ec8b2188b4d1be252269a481651420d6eb5e1"} Nov 02 23:45:04.380571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.380555 5672 scope.go:122] "RemoveContainer" containerID="160373e662a5255ee74af352b9764a1e2227e5241566b773b7c4f651293faefb" Nov 02 23:45:04.380786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.380764 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" event={"ID":"d4ed4836-7860-47fc-be7e-df5a7735b640","Type":"ContainerStarted","Data":"1f3f7df88a9dbe1f03a78e987184679656cca164772c65d9878f6ca2e62faf73"} Nov 02 23:45:04.382625 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.382603 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" event={"ID":"6d12bc18-bd4d-495b-8222-f9682c97db74","Type":"ContainerStarted","Data":"291dac630770c27c4fc8bbaa1d321750d47309d68ad62e70db69602f374d7d37"} Nov 02 23:45:04.384412 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.384393 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" event={"ID":"e1a4d4c2-a470-4945-85a4-2ceec73fd11b","Type":"ContainerStarted","Data":"390a31f96fe390743214644b6a0f4623d995d0d0c4e19ee788c2ba042f913306"} Nov 02 23:45:04.393923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.393901 5672 scope.go:122] "RemoveContainer" containerID="f6352d65eeaac2095ccecfe295a97c73241d108b6d82cb9796a4aa390c828f76" Nov 02 23:45:04.407009 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.406986 5672 scope.go:122] "RemoveContainer" containerID="ac651872be2ad1d4266ab700ca70cfaf496a30c9f560e6bf5472e792b1dde5e5" Nov 02 23:45:04.443208 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:04.443141 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" podStartSLOduration=4.443128405 podStartE2EDuration="4.443128405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:04.439759596 +0000 UTC m=+3097.847071426" watchObservedRunningTime="2026-11-02 23:45:04.443128405 +0000 UTC m=+3097.850440234" Nov 02 23:45:05.393953 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.393929 5672 scope.go:122] "RemoveContainer" containerID="fd2b95f2b477ea3cf2e85905fcec1a9ef21d8986504fed2c25e21c2265353a2c" Nov 02 23:45:05.394412 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:05.394222 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp_odh-ai-gateway-infra(558a2c75-72b2-41cf-b3e4-9d2dc53437f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" Nov 02 23:45:05.395652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.395632 5672 scope.go:122] "RemoveContainer" containerID="e29b22a289baf4674fcdbd41827adec2beed830705a75be23ad06f9ade6c025c" Nov 02 23:45:05.395915 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:05.395895 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894385-wc8cx_odh-ai-gateway-infra(7cef0b79-c23e-45ff-8936-92f4f0ea3708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" Nov 02 23:45:05.397404 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.397382 5672 scope.go:122] "RemoveContainer" containerID="50106a1a4a216da55ff084c34af8a15e27777f01c515b2f8ae7f6663490fc2d5" Nov 02 23:45:05.397658 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:05.397635 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq_odh-ai-gateway-infra(c9f89391-7aab-40c8-95d8-ae30eb9b2f50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" Nov 02 23:45:05.398933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.398910 5672 generic.go:356] "Generic (PLEG): container finished" podID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerID="53c5f0d1a41a298fc2f0aa3013f67ebe884846e860c8ac817780f857cfbf8061" exitCode=60 Nov 02 23:45:05.399024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.398977 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" event={"ID":"517d111c-1dd2-4e24-99ab-ac1556f891ae","Type":"ContainerDied","Data":"53c5f0d1a41a298fc2f0aa3013f67ebe884846e860c8ac817780f857cfbf8061"} Nov 02 23:45:05.399106 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.399025 5672 scope.go:122] "RemoveContainer" containerID="a6b939b33b966ffb613da34c5d886e861569791415ad050a9b51f4fb46b0463c" Nov 02 23:45:05.399238 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.399221 5672 scope.go:122] "RemoveContainer" containerID="53c5f0d1a41a298fc2f0aa3013f67ebe884846e860c8ac817780f857cfbf8061" Nov 02 23:45:05.399488 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:05.399468 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc_odh-ai-gateway-infra(517d111c-1dd2-4e24-99ab-ac1556f891ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" Nov 02 23:45:05.400860 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.400836 5672 generic.go:356] "Generic (PLEG): container finished" podID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerID="d826fec642ade81feea743a31799ac2d4fd185293d631a8568587b5c1b79588b" exitCode=60 Nov 02 23:45:05.400944 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.400897 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" event={"ID":"f7a89c90-63d6-44ab-a0a5-159fa8096ca5","Type":"ContainerDied","Data":"d826fec642ade81feea743a31799ac2d4fd185293d631a8568587b5c1b79588b"} Nov 02 23:45:05.401091 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.401078 5672 scope.go:122] "RemoveContainer" containerID="d826fec642ade81feea743a31799ac2d4fd185293d631a8568587b5c1b79588b" Nov 02 23:45:05.401345 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:05.401319 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn_odh-ai-gateway-infra(f7a89c90-63d6-44ab-a0a5-159fa8096ca5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" Nov 02 23:45:05.402852 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.402824 5672 generic.go:356] "Generic (PLEG): container finished" podID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerID="ffb4c45896560f9e31aadf10489ec8b2188b4d1be252269a481651420d6eb5e1" exitCode=60 Nov 02 23:45:05.402961 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.402868 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" event={"ID":"90894c0e-93e0-4e47-b550-80b4b66f091e","Type":"ContainerDied","Data":"ffb4c45896560f9e31aadf10489ec8b2188b4d1be252269a481651420d6eb5e1"} Nov 02 23:45:05.403107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.403091 5672 scope.go:122] "RemoveContainer" containerID="ffb4c45896560f9e31aadf10489ec8b2188b4d1be252269a481651420d6eb5e1" Nov 02 23:45:05.403391 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:05.403365 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw_odh-ai-gateway-infra(90894c0e-93e0-4e47-b550-80b4b66f091e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" Nov 02 23:45:05.404872 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.404854 5672 generic.go:356] "Generic (PLEG): container finished" podID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerID="1f3f7df88a9dbe1f03a78e987184679656cca164772c65d9878f6ca2e62faf73" exitCode=60 Nov 02 23:45:05.404970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.404944 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" event={"ID":"d4ed4836-7860-47fc-be7e-df5a7735b640","Type":"ContainerDied","Data":"1f3f7df88a9dbe1f03a78e987184679656cca164772c65d9878f6ca2e62faf73"} Nov 02 23:45:05.405122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.405105 5672 scope.go:122] "RemoveContainer" containerID="1f3f7df88a9dbe1f03a78e987184679656cca164772c65d9878f6ca2e62faf73" Nov 02 23:45:05.405369 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:05.405346 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt_odh-ai-gateway-infra(d4ed4836-7860-47fc-be7e-df5a7735b640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" Nov 02 23:45:05.406988 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.406968 5672 generic.go:356] "Generic (PLEG): container finished" podID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerID="291dac630770c27c4fc8bbaa1d321750d47309d68ad62e70db69602f374d7d37" exitCode=60 Nov 02 23:45:05.407062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.407032 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" event={"ID":"6d12bc18-bd4d-495b-8222-f9682c97db74","Type":"ContainerDied","Data":"291dac630770c27c4fc8bbaa1d321750d47309d68ad62e70db69602f374d7d37"} Nov 02 23:45:05.407252 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.407232 5672 scope.go:122] "RemoveContainer" containerID="291dac630770c27c4fc8bbaa1d321750d47309d68ad62e70db69602f374d7d37" Nov 02 23:45:05.407472 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:05.407442 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv_odh-ai-gateway-infra(6d12bc18-bd4d-495b-8222-f9682c97db74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" Nov 02 23:45:05.409024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.409006 5672 generic.go:356] "Generic (PLEG): container finished" podID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerID="390a31f96fe390743214644b6a0f4623d995d0d0c4e19ee788c2ba042f913306" exitCode=60 Nov 02 23:45:05.409126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.409086 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" event={"ID":"e1a4d4c2-a470-4945-85a4-2ceec73fd11b","Type":"ContainerDied","Data":"390a31f96fe390743214644b6a0f4623d995d0d0c4e19ee788c2ba042f913306"} Nov 02 23:45:05.409308 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.409291 5672 scope.go:122] "RemoveContainer" containerID="390a31f96fe390743214644b6a0f4623d995d0d0c4e19ee788c2ba042f913306" Nov 02 23:45:05.409527 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:05.409502 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c_odh-ai-gateway-infra(e1a4d4c2-a470-4945-85a4-2ceec73fd11b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" Nov 02 23:45:05.410883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.410867 5672 scope.go:122] "RemoveContainer" containerID="214db2607ff806d16d5662a0c7cb18fdac70bb778dc7704474040db4cfcca948" Nov 02 23:45:05.411040 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:05.411024 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb_odh-ai-gateway-infra(bdd99463-1bb3-4b55-a805-49dcae325120)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" Nov 02 23:45:05.412906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.412891 5672 scope.go:122] "RemoveContainer" containerID="d7b0432240d6f1181e43cd0efc86a6be21a989f144634345bca0123612963251" Nov 02 23:45:05.425845 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.425814 5672 scope.go:122] "RemoveContainer" containerID="13a72e8ebf06eb65eefaf917287e8caf37f23161b9b256663b85c3708a32c878" Nov 02 23:45:05.439742 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.439723 5672 scope.go:122] "RemoveContainer" containerID="75c49051dbd56b36641d3a8a04bdd500360116f48d87ef1cd1b9017bce161b5b" Nov 02 23:45:05.452305 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.452283 5672 scope.go:122] "RemoveContainer" containerID="c8e6cf8c559e5bbcd2d37b7dcc9e74f54cac2979ce9757a1be62c2f5dd8f4a55" Nov 02 23:45:05.465983 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:05.465955 5672 scope.go:122] "RemoveContainer" containerID="24dba64a7dc394f21ed1d1a3455a481b375f3391744e96210723d8f9de0f7aa8" Nov 02 23:45:06.419241 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:06.419204 5672 scope.go:122] "RemoveContainer" containerID="390a31f96fe390743214644b6a0f4623d995d0d0c4e19ee788c2ba042f913306" Nov 02 23:45:06.419723 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:06.419540 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c_odh-ai-gateway-infra(e1a4d4c2-a470-4945-85a4-2ceec73fd11b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" Nov 02 23:45:06.421013 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:06.420991 5672 scope.go:122] "RemoveContainer" containerID="53c5f0d1a41a298fc2f0aa3013f67ebe884846e860c8ac817780f857cfbf8061" Nov 02 23:45:06.421315 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:06.421289 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc_odh-ai-gateway-infra(517d111c-1dd2-4e24-99ab-ac1556f891ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" Nov 02 23:45:06.422722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:06.422694 5672 scope.go:122] "RemoveContainer" containerID="d826fec642ade81feea743a31799ac2d4fd185293d631a8568587b5c1b79588b" Nov 02 23:45:06.422950 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:06.422929 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn_odh-ai-gateway-infra(f7a89c90-63d6-44ab-a0a5-159fa8096ca5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" Nov 02 23:45:06.424558 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:06.424539 5672 scope.go:122] "RemoveContainer" containerID="ffb4c45896560f9e31aadf10489ec8b2188b4d1be252269a481651420d6eb5e1" Nov 02 23:45:06.424762 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:06.424742 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw_odh-ai-gateway-infra(90894c0e-93e0-4e47-b550-80b4b66f091e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" Nov 02 23:45:06.426447 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:06.426427 5672 scope.go:122] "RemoveContainer" containerID="1f3f7df88a9dbe1f03a78e987184679656cca164772c65d9878f6ca2e62faf73" Nov 02 23:45:06.426649 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:06.426624 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt_odh-ai-gateway-infra(d4ed4836-7860-47fc-be7e-df5a7735b640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" Nov 02 23:45:06.429565 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:06.429547 5672 scope.go:122] "RemoveContainer" containerID="291dac630770c27c4fc8bbaa1d321750d47309d68ad62e70db69602f374d7d37" Nov 02 23:45:06.429714 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:06.429699 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv_odh-ai-gateway-infra(6d12bc18-bd4d-495b-8222-f9682c97db74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" Nov 02 23:45:07.965888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:07.965857 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:45:13.706258 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:13.706218 5672 scope.go:122] "RemoveContainer" containerID="50106a1a4a216da55ff084c34af8a15e27777f01c515b2f8ae7f6663490fc2d5" Nov 02 23:45:14.495687 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:14.495652 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" event={"ID":"c9f89391-7aab-40c8-95d8-ae30eb9b2f50","Type":"ContainerStarted","Data":"0cb747c182b271ad4d04869ead8eae1fd4271ac567f81471b5cd48963f2721ae"} Nov 02 23:45:14.705834 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:14.705806 5672 scope.go:122] "RemoveContainer" containerID="214db2607ff806d16d5662a0c7cb18fdac70bb778dc7704474040db4cfcca948" Nov 02 23:45:14.706010 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:14.705967 5672 scope.go:122] "RemoveContainer" containerID="e29b22a289baf4674fcdbd41827adec2beed830705a75be23ad06f9ade6c025c" Nov 02 23:45:15.504519 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:15.504480 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" event={"ID":"bdd99463-1bb3-4b55-a805-49dcae325120","Type":"ContainerStarted","Data":"f0ed08c6b75922aabe8f0b3ca2c972733eab4c9640adf5186b6ad44c1da31dd6"} Nov 02 23:45:15.506439 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:15.506410 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" event={"ID":"7cef0b79-c23e-45ff-8936-92f4f0ea3708","Type":"ContainerStarted","Data":"1269d7a589797e6ce5c42e1c9a3e0c84dc8d517c4eba8963e7cfd1ab0fd3ee0c"} Nov 02 23:45:15.508130 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:15.508106 5672 generic.go:356] "Generic (PLEG): container finished" podID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerID="0cb747c182b271ad4d04869ead8eae1fd4271ac567f81471b5cd48963f2721ae" exitCode=60 Nov 02 23:45:15.508265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:15.508179 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" event={"ID":"c9f89391-7aab-40c8-95d8-ae30eb9b2f50","Type":"ContainerDied","Data":"0cb747c182b271ad4d04869ead8eae1fd4271ac567f81471b5cd48963f2721ae"} Nov 02 23:45:15.508265 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:15.508238 5672 scope.go:122] "RemoveContainer" containerID="50106a1a4a216da55ff084c34af8a15e27777f01c515b2f8ae7f6663490fc2d5" Nov 02 23:45:15.508497 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:15.508483 5672 scope.go:122] "RemoveContainer" containerID="0cb747c182b271ad4d04869ead8eae1fd4271ac567f81471b5cd48963f2721ae" Nov 02 23:45:15.508729 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:15.508711 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq_odh-ai-gateway-infra(c9f89391-7aab-40c8-95d8-ae30eb9b2f50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" Nov 02 23:45:15.612844 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:15.612801 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq"] Nov 02 23:45:15.705972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:15.705939 5672 scope.go:122] "RemoveContainer" containerID="fd2b95f2b477ea3cf2e85905fcec1a9ef21d8986504fed2c25e21c2265353a2c" Nov 02 23:45:15.706114 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:15.706083 5672 scope.go:122] "RemoveContainer" containerID="390a31f96fe390743214644b6a0f4623d995d0d0c4e19ee788c2ba042f913306" Nov 02 23:45:16.516382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.516342 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" event={"ID":"e1a4d4c2-a470-4945-85a4-2ceec73fd11b","Type":"ContainerStarted","Data":"e321b70411332d8cf0069b3da919e91a3f9403fe2a5ddff03b0684ef55bec62e"} Nov 02 23:45:16.518320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.518288 5672 generic.go:356] "Generic (PLEG): container finished" podID="bdd99463-1bb3-4b55-a805-49dcae325120" containerID="f0ed08c6b75922aabe8f0b3ca2c972733eab4c9640adf5186b6ad44c1da31dd6" exitCode=60 Nov 02 23:45:16.518452 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.518364 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" event={"ID":"bdd99463-1bb3-4b55-a805-49dcae325120","Type":"ContainerDied","Data":"f0ed08c6b75922aabe8f0b3ca2c972733eab4c9640adf5186b6ad44c1da31dd6"} Nov 02 23:45:16.518452 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.518418 5672 scope.go:122] "RemoveContainer" containerID="214db2607ff806d16d5662a0c7cb18fdac70bb778dc7704474040db4cfcca948" Nov 02 23:45:16.518717 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.518696 5672 scope.go:122] "RemoveContainer" containerID="f0ed08c6b75922aabe8f0b3ca2c972733eab4c9640adf5186b6ad44c1da31dd6" Nov 02 23:45:16.518991 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:16.518971 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb_odh-ai-gateway-infra(bdd99463-1bb3-4b55-a805-49dcae325120)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" Nov 02 23:45:16.520458 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.520435 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" event={"ID":"558a2c75-72b2-41cf-b3e4-9d2dc53437f3","Type":"ContainerStarted","Data":"f9fa732ddc3c2c7051af6b6fcb3ee45b1906d7f19ee967245a634d5631fad755"} Nov 02 23:45:16.522310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.522290 5672 generic.go:356] "Generic (PLEG): container finished" podID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerID="1269d7a589797e6ce5c42e1c9a3e0c84dc8d517c4eba8963e7cfd1ab0fd3ee0c" exitCode=60 Nov 02 23:45:16.522420 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.522360 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" event={"ID":"7cef0b79-c23e-45ff-8936-92f4f0ea3708","Type":"ContainerDied","Data":"1269d7a589797e6ce5c42e1c9a3e0c84dc8d517c4eba8963e7cfd1ab0fd3ee0c"} Nov 02 23:45:16.522644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.522627 5672 scope.go:122] "RemoveContainer" containerID="1269d7a589797e6ce5c42e1c9a3e0c84dc8d517c4eba8963e7cfd1ab0fd3ee0c" Nov 02 23:45:16.522880 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:16.522856 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894385-wc8cx_odh-ai-gateway-infra(7cef0b79-c23e-45ff-8936-92f4f0ea3708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" Nov 02 23:45:16.532419 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.532398 5672 scope.go:122] "RemoveContainer" containerID="e29b22a289baf4674fcdbd41827adec2beed830705a75be23ad06f9ade6c025c" Nov 02 23:45:16.622923 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.622890 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb"] Nov 02 23:45:16.678966 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.678938 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" Nov 02 23:45:16.727112 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.727078 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx"] Nov 02 23:45:16.770848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.770767 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-maas-api-service-ca\") pod \"c9f89391-7aab-40c8-95d8-ae30eb9b2f50\" (UID: \"c9f89391-7aab-40c8-95d8-ae30eb9b2f50\") " Nov 02 23:45:16.771005 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.770945 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-kube-api-access-tq92h\" (UniqueName: \"kubernetes.io/projected/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-kube-api-access-tq92h\") pod \"c9f89391-7aab-40c8-95d8-ae30eb9b2f50\" (UID: \"c9f89391-7aab-40c8-95d8-ae30eb9b2f50\") " Nov 02 23:45:16.771143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.771121 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-maas-api-service-ca" pod "c9f89391-7aab-40c8-95d8-ae30eb9b2f50" (UID: "c9f89391-7aab-40c8-95d8-ae30eb9b2f50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:16.771309 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.771294 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:16.772979 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.772961 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-kube-api-access-tq92h" pod "c9f89391-7aab-40c8-95d8-ae30eb9b2f50" (UID: "c9f89391-7aab-40c8-95d8-ae30eb9b2f50"). InnerVolumeSpecName "kube-api-access-tq92h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:16.872839 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:16.872805 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq92h\" (UniqueName: \"kubernetes.io/projected/c9f89391-7aab-40c8-95d8-ae30eb9b2f50-kube-api-access-tq92h\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:17.536697 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.536656 5672 generic.go:356] "Generic (PLEG): container finished" podID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerID="f9fa732ddc3c2c7051af6b6fcb3ee45b1906d7f19ee967245a634d5631fad755" exitCode=60 Nov 02 23:45:17.537112 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.536718 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" event={"ID":"558a2c75-72b2-41cf-b3e4-9d2dc53437f3","Type":"ContainerDied","Data":"f9fa732ddc3c2c7051af6b6fcb3ee45b1906d7f19ee967245a634d5631fad755"} Nov 02 23:45:17.537112 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.536805 5672 scope.go:122] "RemoveContainer" containerID="fd2b95f2b477ea3cf2e85905fcec1a9ef21d8986504fed2c25e21c2265353a2c" Nov 02 23:45:17.537317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.537300 5672 scope.go:122] "RemoveContainer" containerID="f9fa732ddc3c2c7051af6b6fcb3ee45b1906d7f19ee967245a634d5631fad755" Nov 02 23:45:17.537673 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:17.537639 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp_odh-ai-gateway-infra(558a2c75-72b2-41cf-b3e4-9d2dc53437f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" Nov 02 23:45:17.540144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.540124 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" Nov 02 23:45:17.540303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.540198 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq" event={"ID":"c9f89391-7aab-40c8-95d8-ae30eb9b2f50","Type":"ContainerDied","Data":"da3a1bb08cedcf5db58c077ff9a233e2f77afb3d37e3c11e25d881b092501a12"} Nov 02 23:45:17.542612 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.542585 5672 generic.go:356] "Generic (PLEG): container finished" podID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerID="e321b70411332d8cf0069b3da919e91a3f9403fe2a5ddff03b0684ef55bec62e" exitCode=60 Nov 02 23:45:17.542723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.542628 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" event={"ID":"e1a4d4c2-a470-4945-85a4-2ceec73fd11b","Type":"ContainerDied","Data":"e321b70411332d8cf0069b3da919e91a3f9403fe2a5ddff03b0684ef55bec62e"} Nov 02 23:45:17.543006 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.542988 5672 scope.go:122] "RemoveContainer" containerID="e321b70411332d8cf0069b3da919e91a3f9403fe2a5ddff03b0684ef55bec62e" Nov 02 23:45:17.543548 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:17.543380 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c_odh-ai-gateway-infra(e1a4d4c2-a470-4945-85a4-2ceec73fd11b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" Nov 02 23:45:17.553529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.553513 5672 scope.go:122] "RemoveContainer" containerID="0cb747c182b271ad4d04869ead8eae1fd4271ac567f81471b5cd48963f2721ae" Nov 02 23:45:17.565057 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.565037 5672 scope.go:122] "RemoveContainer" containerID="390a31f96fe390743214644b6a0f4623d995d0d0c4e19ee788c2ba042f913306" Nov 02 23:45:17.639549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.639198 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c"] Nov 02 23:45:17.705832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.705806 5672 scope.go:122] "RemoveContainer" containerID="291dac630770c27c4fc8bbaa1d321750d47309d68ad62e70db69602f374d7d37" Nov 02 23:45:17.706141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.706114 5672 scope.go:122] "RemoveContainer" containerID="1f3f7df88a9dbe1f03a78e987184679656cca164772c65d9878f6ca2e62faf73" Nov 02 23:45:17.706398 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.706377 5672 scope.go:122] "RemoveContainer" containerID="ffb4c45896560f9e31aadf10489ec8b2188b4d1be252269a481651420d6eb5e1" Nov 02 23:45:17.742552 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.742529 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" Nov 02 23:45:17.748890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.748864 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" Nov 02 23:45:17.782064 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.782039 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cef0b79-c23e-45ff-8936-92f4f0ea3708-kube-api-access-2kcjs\" (UniqueName: \"kubernetes.io/projected/7cef0b79-c23e-45ff-8936-92f4f0ea3708-kube-api-access-2kcjs\") pod \"7cef0b79-c23e-45ff-8936-92f4f0ea3708\" (UID: \"7cef0b79-c23e-45ff-8936-92f4f0ea3708\") " Nov 02 23:45:17.782231 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.782102 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cef0b79-c23e-45ff-8936-92f4f0ea3708-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef0b79-c23e-45ff-8936-92f4f0ea3708-maas-api-service-ca\") pod \"7cef0b79-c23e-45ff-8936-92f4f0ea3708\" (UID: \"7cef0b79-c23e-45ff-8936-92f4f0ea3708\") " Nov 02 23:45:17.782306 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.782250 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd99463-1bb3-4b55-a805-49dcae325120-kube-api-access-pfdxg\" (UniqueName: \"kubernetes.io/projected/bdd99463-1bb3-4b55-a805-49dcae325120-kube-api-access-pfdxg\") pod \"bdd99463-1bb3-4b55-a805-49dcae325120\" (UID: \"bdd99463-1bb3-4b55-a805-49dcae325120\") " Nov 02 23:45:17.782306 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.782285 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd99463-1bb3-4b55-a805-49dcae325120-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd99463-1bb3-4b55-a805-49dcae325120-maas-api-service-ca\") pod \"bdd99463-1bb3-4b55-a805-49dcae325120\" (UID: \"bdd99463-1bb3-4b55-a805-49dcae325120\") " Nov 02 23:45:17.782526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.782481 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cef0b79-c23e-45ff-8936-92f4f0ea3708-maas-api-service-ca" pod "7cef0b79-c23e-45ff-8936-92f4f0ea3708" (UID: "7cef0b79-c23e-45ff-8936-92f4f0ea3708"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:17.783506 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.782845 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd99463-1bb3-4b55-a805-49dcae325120-maas-api-service-ca" pod "bdd99463-1bb3-4b55-a805-49dcae325120" (UID: "bdd99463-1bb3-4b55-a805-49dcae325120"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:17.783506 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.782942 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cef0b79-c23e-45ff-8936-92f4f0ea3708-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:17.783506 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.782959 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd99463-1bb3-4b55-a805-49dcae325120-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:17.784507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.784486 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cef0b79-c23e-45ff-8936-92f4f0ea3708-kube-api-access-2kcjs" pod "7cef0b79-c23e-45ff-8936-92f4f0ea3708" (UID: "7cef0b79-c23e-45ff-8936-92f4f0ea3708"). InnerVolumeSpecName "kube-api-access-2kcjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:17.785207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.785117 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd99463-1bb3-4b55-a805-49dcae325120-kube-api-access-pfdxg" pod "bdd99463-1bb3-4b55-a805-49dcae325120" (UID: "bdd99463-1bb3-4b55-a805-49dcae325120"). InnerVolumeSpecName "kube-api-access-pfdxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:17.883637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.883605 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfdxg\" (UniqueName: \"kubernetes.io/projected/bdd99463-1bb3-4b55-a805-49dcae325120-kube-api-access-pfdxg\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:17.883637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:17.883632 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kcjs\" (UniqueName: \"kubernetes.io/projected/7cef0b79-c23e-45ff-8936-92f4f0ea3708-kube-api-access-2kcjs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:18.055424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.055398 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp"] Nov 02 23:45:18.072234 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.072207 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq"] Nov 02 23:45:18.077336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.077310 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894385-sg9hq"] Nov 02 23:45:18.553212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.553181 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" Nov 02 23:45:18.553212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.553198 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx" event={"ID":"7cef0b79-c23e-45ff-8936-92f4f0ea3708","Type":"ContainerDied","Data":"cd7437933f420109b8466984f0da3d76aa1974cafd3eab4c5decef51381a00e1"} Nov 02 23:45:18.553814 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.553274 5672 scope.go:122] "RemoveContainer" containerID="1269d7a589797e6ce5c42e1c9a3e0c84dc8d517c4eba8963e7cfd1ab0fd3ee0c" Nov 02 23:45:18.557341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.557318 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" event={"ID":"90894c0e-93e0-4e47-b550-80b4b66f091e","Type":"ContainerStarted","Data":"bfba5aace90adc64b6d72ac2fae3fb002598a505a3e60c551244bf7973539424"} Nov 02 23:45:18.560336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.560314 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" event={"ID":"d4ed4836-7860-47fc-be7e-df5a7735b640","Type":"ContainerStarted","Data":"a47e8719b5a4f1ace059d18d388c8b5d59b37ee17fb1d2e056d217fa59abe090"} Nov 02 23:45:18.562477 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.562448 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" event={"ID":"6d12bc18-bd4d-495b-8222-f9682c97db74","Type":"ContainerStarted","Data":"57103ef78cac33071abe410630897a7cf5eff1e48b008647683fc2c1bd636761"} Nov 02 23:45:18.566464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.566443 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" Nov 02 23:45:18.566571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.566473 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb" event={"ID":"bdd99463-1bb3-4b55-a805-49dcae325120","Type":"ContainerDied","Data":"e1331a436c080dabf4de1bdce10a9c8c4b16a8afefd5f1f349dcb6ab9d8a971d"} Nov 02 23:45:18.566571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.566521 5672 scope.go:122] "RemoveContainer" containerID="f0ed08c6b75922aabe8f0b3ca2c972733eab4c9640adf5186b6ad44c1da31dd6" Nov 02 23:45:18.709603 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.708664 5672 scope.go:122] "RemoveContainer" containerID="53c5f0d1a41a298fc2f0aa3013f67ebe884846e860c8ac817780f857cfbf8061" Nov 02 23:45:18.712277 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.711789 5672 scope.go:122] "RemoveContainer" containerID="d826fec642ade81feea743a31799ac2d4fd185293d631a8568587b5c1b79588b" Nov 02 23:45:18.726043 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.726001 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" path="/var/lib/kubelet/pods/c9f89391-7aab-40c8-95d8-ae30eb9b2f50/volumes" Nov 02 23:45:18.932636 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.932610 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" Nov 02 23:45:18.946890 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.946864 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" Nov 02 23:45:18.998217 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.998182 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-maas-api-service-ca\") pod \"558a2c75-72b2-41cf-b3e4-9d2dc53437f3\" (UID: \"558a2c75-72b2-41cf-b3e4-9d2dc53437f3\") " Nov 02 23:45:18.998402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.998276 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-kube-api-access-slrtb\" (UniqueName: \"kubernetes.io/projected/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-kube-api-access-slrtb\") pod \"558a2c75-72b2-41cf-b3e4-9d2dc53437f3\" (UID: \"558a2c75-72b2-41cf-b3e4-9d2dc53437f3\") " Nov 02 23:45:18.998402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.998304 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-kube-api-access-vml54\" (UniqueName: \"kubernetes.io/projected/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-kube-api-access-vml54\") pod \"e1a4d4c2-a470-4945-85a4-2ceec73fd11b\" (UID: \"e1a4d4c2-a470-4945-85a4-2ceec73fd11b\") " Nov 02 23:45:18.998402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.998360 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-maas-api-service-ca\") pod \"e1a4d4c2-a470-4945-85a4-2ceec73fd11b\" (UID: \"e1a4d4c2-a470-4945-85a4-2ceec73fd11b\") " Nov 02 23:45:18.998719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.998686 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-maas-api-service-ca" pod "558a2c75-72b2-41cf-b3e4-9d2dc53437f3" (UID: "558a2c75-72b2-41cf-b3e4-9d2dc53437f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:18.998817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.998795 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-maas-api-service-ca" pod "e1a4d4c2-a470-4945-85a4-2ceec73fd11b" (UID: "e1a4d4c2-a470-4945-85a4-2ceec73fd11b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:18.998880 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:18.998869 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:19.000526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.000505 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-kube-api-access-vml54" pod "e1a4d4c2-a470-4945-85a4-2ceec73fd11b" (UID: "e1a4d4c2-a470-4945-85a4-2ceec73fd11b"). InnerVolumeSpecName "kube-api-access-vml54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:19.000613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.000533 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-kube-api-access-slrtb" pod "558a2c75-72b2-41cf-b3e4-9d2dc53437f3" (UID: "558a2c75-72b2-41cf-b3e4-9d2dc53437f3"). InnerVolumeSpecName "kube-api-access-slrtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:19.046294 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.046257 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx"] Nov 02 23:45:19.049171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.049134 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894385-wc8cx"] Nov 02 23:45:19.100450 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.100381 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slrtb\" (UniqueName: \"kubernetes.io/projected/558a2c75-72b2-41cf-b3e4-9d2dc53437f3-kube-api-access-slrtb\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:19.100450 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.100407 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vml54\" (UniqueName: \"kubernetes.io/projected/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-kube-api-access-vml54\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:19.100450 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.100418 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a4d4c2-a470-4945-85a4-2ceec73fd11b-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:19.358283 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.358225 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb"] Nov 02 23:45:19.362218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.362187 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894385-jq2wb"] Nov 02 23:45:19.598929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.598891 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" event={"ID":"517d111c-1dd2-4e24-99ab-ac1556f891ae","Type":"ContainerStarted","Data":"3b9b9942c195399706bbbf90e35f590942b5173db1d810c1b392ba06c235633c"} Nov 02 23:45:19.600885 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.600853 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" event={"ID":"f7a89c90-63d6-44ab-a0a5-159fa8096ca5","Type":"ContainerStarted","Data":"65b181d26f8462769e265a68a7521171821777d12b1e46452ec745d1160a93c9"} Nov 02 23:45:19.602851 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.602826 5672 generic.go:356] "Generic (PLEG): container finished" podID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerID="bfba5aace90adc64b6d72ac2fae3fb002598a505a3e60c551244bf7973539424" exitCode=60 Nov 02 23:45:19.602971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.602898 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" event={"ID":"90894c0e-93e0-4e47-b550-80b4b66f091e","Type":"ContainerDied","Data":"bfba5aace90adc64b6d72ac2fae3fb002598a505a3e60c551244bf7973539424"} Nov 02 23:45:19.602971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.602960 5672 scope.go:122] "RemoveContainer" containerID="ffb4c45896560f9e31aadf10489ec8b2188b4d1be252269a481651420d6eb5e1" Nov 02 23:45:19.603274 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.603251 5672 scope.go:122] "RemoveContainer" containerID="bfba5aace90adc64b6d72ac2fae3fb002598a505a3e60c551244bf7973539424" Nov 02 23:45:19.603554 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:19.603526 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw_odh-ai-gateway-infra(90894c0e-93e0-4e47-b550-80b4b66f091e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" Nov 02 23:45:19.604843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.604821 5672 generic.go:356] "Generic (PLEG): container finished" podID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerID="a47e8719b5a4f1ace059d18d388c8b5d59b37ee17fb1d2e056d217fa59abe090" exitCode=60 Nov 02 23:45:19.604949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.604900 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" event={"ID":"d4ed4836-7860-47fc-be7e-df5a7735b640","Type":"ContainerDied","Data":"a47e8719b5a4f1ace059d18d388c8b5d59b37ee17fb1d2e056d217fa59abe090"} Nov 02 23:45:19.605338 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.605314 5672 scope.go:122] "RemoveContainer" containerID="a47e8719b5a4f1ace059d18d388c8b5d59b37ee17fb1d2e056d217fa59abe090" Nov 02 23:45:19.605604 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:19.605579 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt_odh-ai-gateway-infra(d4ed4836-7860-47fc-be7e-df5a7735b640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" Nov 02 23:45:19.606786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.606767 5672 generic.go:356] "Generic (PLEG): container finished" podID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerID="57103ef78cac33071abe410630897a7cf5eff1e48b008647683fc2c1bd636761" exitCode=60 Nov 02 23:45:19.606877 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.606834 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" event={"ID":"6d12bc18-bd4d-495b-8222-f9682c97db74","Type":"ContainerDied","Data":"57103ef78cac33071abe410630897a7cf5eff1e48b008647683fc2c1bd636761"} Nov 02 23:45:19.607109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.607092 5672 scope.go:122] "RemoveContainer" containerID="57103ef78cac33071abe410630897a7cf5eff1e48b008647683fc2c1bd636761" Nov 02 23:45:19.607343 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:19.607318 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv_odh-ai-gateway-infra(6d12bc18-bd4d-495b-8222-f9682c97db74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" Nov 02 23:45:19.608334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.608281 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" event={"ID":"e1a4d4c2-a470-4945-85a4-2ceec73fd11b","Type":"ContainerDied","Data":"67fcaf53cd90acf7ab8ba2b1d0ca712f936b735621f1e01db3fc35ce7d987737"} Nov 02 23:45:19.608334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.608297 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c" Nov 02 23:45:19.610926 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.610903 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" event={"ID":"558a2c75-72b2-41cf-b3e4-9d2dc53437f3","Type":"ContainerDied","Data":"4abfdd41c4a04d1c0a9817509b81cf2ad7517d483de155461629a08b6f1a0108"} Nov 02 23:45:19.611041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.610960 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp" Nov 02 23:45:19.624405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.624388 5672 scope.go:122] "RemoveContainer" containerID="1f3f7df88a9dbe1f03a78e987184679656cca164772c65d9878f6ca2e62faf73" Nov 02 23:45:19.639757 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.639740 5672 scope.go:122] "RemoveContainer" containerID="291dac630770c27c4fc8bbaa1d321750d47309d68ad62e70db69602f374d7d37" Nov 02 23:45:19.652225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.652206 5672 scope.go:122] "RemoveContainer" containerID="e321b70411332d8cf0069b3da919e91a3f9403fe2a5ddff03b0684ef55bec62e" Nov 02 23:45:19.660643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.660626 5672 scope.go:122] "RemoveContainer" containerID="f9fa732ddc3c2c7051af6b6fcb3ee45b1906d7f19ee967245a634d5631fad755" Nov 02 23:45:19.719251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.719221 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv"] Nov 02 23:45:19.813748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.813713 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c"] Nov 02 23:45:19.817240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.817216 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894385-zz69c"] Nov 02 23:45:19.845124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.845095 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw"] Nov 02 23:45:19.917518 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.917478 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp"] Nov 02 23:45:19.924925 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:19.924902 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894385-pjpkp"] Nov 02 23:45:20.265292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.265251 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt"] Nov 02 23:45:20.620990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.620904 5672 generic.go:356] "Generic (PLEG): container finished" podID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerID="3b9b9942c195399706bbbf90e35f590942b5173db1d810c1b392ba06c235633c" exitCode=60 Nov 02 23:45:20.620990 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.620973 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" event={"ID":"517d111c-1dd2-4e24-99ab-ac1556f891ae","Type":"ContainerDied","Data":"3b9b9942c195399706bbbf90e35f590942b5173db1d810c1b392ba06c235633c"} Nov 02 23:45:20.621475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.621038 5672 scope.go:122] "RemoveContainer" containerID="53c5f0d1a41a298fc2f0aa3013f67ebe884846e860c8ac817780f857cfbf8061" Nov 02 23:45:20.621475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.621436 5672 scope.go:122] "RemoveContainer" containerID="3b9b9942c195399706bbbf90e35f590942b5173db1d810c1b392ba06c235633c" Nov 02 23:45:20.621767 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:20.621742 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc_odh-ai-gateway-infra(517d111c-1dd2-4e24-99ab-ac1556f891ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" Nov 02 23:45:20.622930 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.622910 5672 generic.go:356] "Generic (PLEG): container finished" podID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerID="65b181d26f8462769e265a68a7521171821777d12b1e46452ec745d1160a93c9" exitCode=60 Nov 02 23:45:20.623029 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.622979 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" event={"ID":"f7a89c90-63d6-44ab-a0a5-159fa8096ca5","Type":"ContainerDied","Data":"65b181d26f8462769e265a68a7521171821777d12b1e46452ec745d1160a93c9"} Nov 02 23:45:20.623302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.623284 5672 scope.go:122] "RemoveContainer" containerID="65b181d26f8462769e265a68a7521171821777d12b1e46452ec745d1160a93c9" Nov 02 23:45:20.623537 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:20.623519 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn_odh-ai-gateway-infra(f7a89c90-63d6-44ab-a0a5-159fa8096ca5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" Nov 02 23:45:20.636686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.636660 5672 scope.go:122] "RemoveContainer" containerID="d826fec642ade81feea743a31799ac2d4fd185293d631a8568587b5c1b79588b" Nov 02 23:45:20.717615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.717580 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" path="/var/lib/kubelet/pods/558a2c75-72b2-41cf-b3e4-9d2dc53437f3/volumes" Nov 02 23:45:20.718139 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.718116 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" path="/var/lib/kubelet/pods/7cef0b79-c23e-45ff-8936-92f4f0ea3708/volumes" Nov 02 23:45:20.718646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.718627 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" path="/var/lib/kubelet/pods/bdd99463-1bb3-4b55-a805-49dcae325120/volumes" Nov 02 23:45:20.719697 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.719679 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" path="/var/lib/kubelet/pods/e1a4d4c2-a470-4945-85a4-2ceec73fd11b/volumes" Nov 02 23:45:20.719998 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.719983 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc"] Nov 02 23:45:20.786012 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.785984 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" Nov 02 23:45:20.819638 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.819600 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90894c0e-93e0-4e47-b550-80b4b66f091e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90894c0e-93e0-4e47-b550-80b4b66f091e-maas-api-service-ca\") pod \"90894c0e-93e0-4e47-b550-80b4b66f091e\" (UID: \"90894c0e-93e0-4e47-b550-80b4b66f091e\") " Nov 02 23:45:20.819786 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.819641 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90894c0e-93e0-4e47-b550-80b4b66f091e-kube-api-access-d99vz\" (UniqueName: \"kubernetes.io/projected/90894c0e-93e0-4e47-b550-80b4b66f091e-kube-api-access-d99vz\") pod \"90894c0e-93e0-4e47-b550-80b4b66f091e\" (UID: \"90894c0e-93e0-4e47-b550-80b4b66f091e\") " Nov 02 23:45:20.819965 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.819942 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90894c0e-93e0-4e47-b550-80b4b66f091e-maas-api-service-ca" pod "90894c0e-93e0-4e47-b550-80b4b66f091e" (UID: "90894c0e-93e0-4e47-b550-80b4b66f091e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:20.820232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.820215 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90894c0e-93e0-4e47-b550-80b4b66f091e-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.822257 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.822235 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90894c0e-93e0-4e47-b550-80b4b66f091e-kube-api-access-d99vz" pod "90894c0e-93e0-4e47-b550-80b4b66f091e" (UID: "90894c0e-93e0-4e47-b550-80b4b66f091e"). InnerVolumeSpecName "kube-api-access-d99vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:20.857894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.857870 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" Nov 02 23:45:20.862571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.862551 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" Nov 02 23:45:20.921497 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.921414 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ed4836-7860-47fc-be7e-df5a7735b640-kube-api-access-gbgjk\" (UniqueName: \"kubernetes.io/projected/d4ed4836-7860-47fc-be7e-df5a7735b640-kube-api-access-gbgjk\") pod \"d4ed4836-7860-47fc-be7e-df5a7735b640\" (UID: \"d4ed4836-7860-47fc-be7e-df5a7735b640\") " Nov 02 23:45:20.921497 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.921484 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d12bc18-bd4d-495b-8222-f9682c97db74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d12bc18-bd4d-495b-8222-f9682c97db74-maas-api-service-ca\") pod \"6d12bc18-bd4d-495b-8222-f9682c97db74\" (UID: \"6d12bc18-bd4d-495b-8222-f9682c97db74\") " Nov 02 23:45:20.921718 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.921519 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ed4836-7860-47fc-be7e-df5a7735b640-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ed4836-7860-47fc-be7e-df5a7735b640-maas-api-service-ca\") pod \"d4ed4836-7860-47fc-be7e-df5a7735b640\" (UID: \"d4ed4836-7860-47fc-be7e-df5a7735b640\") " Nov 02 23:45:20.921774 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.921730 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d12bc18-bd4d-495b-8222-f9682c97db74-kube-api-access-g89zp\" (UniqueName: \"kubernetes.io/projected/6d12bc18-bd4d-495b-8222-f9682c97db74-kube-api-access-g89zp\") pod \"6d12bc18-bd4d-495b-8222-f9682c97db74\" (UID: \"6d12bc18-bd4d-495b-8222-f9682c97db74\") " Nov 02 23:45:20.921894 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.921865 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d12bc18-bd4d-495b-8222-f9682c97db74-maas-api-service-ca" pod "6d12bc18-bd4d-495b-8222-f9682c97db74" (UID: "6d12bc18-bd4d-495b-8222-f9682c97db74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:20.921955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.921895 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ed4836-7860-47fc-be7e-df5a7735b640-maas-api-service-ca" pod "d4ed4836-7860-47fc-be7e-df5a7735b640" (UID: "d4ed4836-7860-47fc-be7e-df5a7735b640"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:20.922212 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.922193 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d99vz\" (UniqueName: \"kubernetes.io/projected/90894c0e-93e0-4e47-b550-80b4b66f091e-kube-api-access-d99vz\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.922267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.922215 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d12bc18-bd4d-495b-8222-f9682c97db74-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.922267 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.922225 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ed4836-7860-47fc-be7e-df5a7735b640-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:20.923484 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.923461 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ed4836-7860-47fc-be7e-df5a7735b640-kube-api-access-gbgjk" pod "d4ed4836-7860-47fc-be7e-df5a7735b640" (UID: "d4ed4836-7860-47fc-be7e-df5a7735b640"). InnerVolumeSpecName "kube-api-access-gbgjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:20.923634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:20.923616 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d12bc18-bd4d-495b-8222-f9682c97db74-kube-api-access-g89zp" pod "6d12bc18-bd4d-495b-8222-f9682c97db74" (UID: "6d12bc18-bd4d-495b-8222-f9682c97db74"). InnerVolumeSpecName "kube-api-access-g89zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:21.023037 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.023005 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g89zp\" (UniqueName: \"kubernetes.io/projected/6d12bc18-bd4d-495b-8222-f9682c97db74-kube-api-access-g89zp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:21.023037 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.023035 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbgjk\" (UniqueName: \"kubernetes.io/projected/d4ed4836-7860-47fc-be7e-df5a7735b640-kube-api-access-gbgjk\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:21.037118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.037080 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn"] Nov 02 23:45:21.638478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.638437 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" event={"ID":"90894c0e-93e0-4e47-b550-80b4b66f091e","Type":"ContainerDied","Data":"9a11ee4eb2cde0999c86c6e34eebfa4bcf3c0c37a03b8d7ee1214319624d3fd9"} Nov 02 23:45:21.638876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.638447 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw" Nov 02 23:45:21.638876 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.638525 5672 scope.go:122] "RemoveContainer" containerID="bfba5aace90adc64b6d72ac2fae3fb002598a505a3e60c551244bf7973539424" Nov 02 23:45:21.640594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.640567 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" event={"ID":"d4ed4836-7860-47fc-be7e-df5a7735b640","Type":"ContainerDied","Data":"05a75e2da61d530f4b637c492d301ad954f507a097af1249a5f24c1fb6f8e113"} Nov 02 23:45:21.640714 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.640671 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt" Nov 02 23:45:21.650954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.650927 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" event={"ID":"6d12bc18-bd4d-495b-8222-f9682c97db74","Type":"ContainerDied","Data":"854167b5f751ea316f53f92dc8739b736aed32ea580d46d2c480bd556e4d6f04"} Nov 02 23:45:21.650954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.650946 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv" Nov 02 23:45:21.651096 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.651025 5672 scope.go:122] "RemoveContainer" containerID="a47e8719b5a4f1ace059d18d388c8b5d59b37ee17fb1d2e056d217fa59abe090" Nov 02 23:45:21.673327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.673302 5672 scope.go:122] "RemoveContainer" containerID="57103ef78cac33071abe410630897a7cf5eff1e48b008647683fc2c1bd636761" Nov 02 23:45:21.814530 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.814502 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" Nov 02 23:45:21.816485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.816464 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" Nov 02 23:45:21.935236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.935129 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-kube-api-access-qs99g\" (UniqueName: \"kubernetes.io/projected/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-kube-api-access-qs99g\") pod \"f7a89c90-63d6-44ab-a0a5-159fa8096ca5\" (UID: \"f7a89c90-63d6-44ab-a0a5-159fa8096ca5\") " Nov 02 23:45:21.935418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.935314 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-maas-api-service-ca\") pod \"f7a89c90-63d6-44ab-a0a5-159fa8096ca5\" (UID: \"f7a89c90-63d6-44ab-a0a5-159fa8096ca5\") " Nov 02 23:45:21.935418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.935374 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517d111c-1dd2-4e24-99ab-ac1556f891ae-kube-api-access-mkjmv\" (UniqueName: \"kubernetes.io/projected/517d111c-1dd2-4e24-99ab-ac1556f891ae-kube-api-access-mkjmv\") pod \"517d111c-1dd2-4e24-99ab-ac1556f891ae\" (UID: \"517d111c-1dd2-4e24-99ab-ac1556f891ae\") " Nov 02 23:45:21.935418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.935400 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/517d111c-1dd2-4e24-99ab-ac1556f891ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/517d111c-1dd2-4e24-99ab-ac1556f891ae-maas-api-service-ca\") pod \"517d111c-1dd2-4e24-99ab-ac1556f891ae\" (UID: \"517d111c-1dd2-4e24-99ab-ac1556f891ae\") " Nov 02 23:45:21.935820 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.935779 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517d111c-1dd2-4e24-99ab-ac1556f891ae-maas-api-service-ca" pod "517d111c-1dd2-4e24-99ab-ac1556f891ae" (UID: "517d111c-1dd2-4e24-99ab-ac1556f891ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:21.935972 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.935923 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/517d111c-1dd2-4e24-99ab-ac1556f891ae-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:21.936282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.936253 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-maas-api-service-ca" pod "f7a89c90-63d6-44ab-a0a5-159fa8096ca5" (UID: "f7a89c90-63d6-44ab-a0a5-159fa8096ca5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:21.937771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.937740 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-kube-api-access-qs99g" pod "f7a89c90-63d6-44ab-a0a5-159fa8096ca5" (UID: "f7a89c90-63d6-44ab-a0a5-159fa8096ca5"). InnerVolumeSpecName "kube-api-access-qs99g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:21.938285 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.938263 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517d111c-1dd2-4e24-99ab-ac1556f891ae-kube-api-access-mkjmv" pod "517d111c-1dd2-4e24-99ab-ac1556f891ae" (UID: "517d111c-1dd2-4e24-99ab-ac1556f891ae"). InnerVolumeSpecName "kube-api-access-mkjmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:21.963404 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.963379 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt"] Nov 02 23:45:21.967552 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:21.967526 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894385-rvspt"] Nov 02 23:45:22.037230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.037194 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs99g\" (UniqueName: \"kubernetes.io/projected/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-kube-api-access-qs99g\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:22.037230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.037227 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a89c90-63d6-44ab-a0a5-159fa8096ca5-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:22.037418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.037243 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkjmv\" (UniqueName: \"kubernetes.io/projected/517d111c-1dd2-4e24-99ab-ac1556f891ae-kube-api-access-mkjmv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:22.067784 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.067752 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv"] Nov 02 23:45:22.071761 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.071730 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894385-ffmzv"] Nov 02 23:45:22.175002 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.174970 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw"] Nov 02 23:45:22.181615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.181589 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894385-2rtmw"] Nov 02 23:45:22.663821 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.663785 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" Nov 02 23:45:22.664350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.663790 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc" event={"ID":"517d111c-1dd2-4e24-99ab-ac1556f891ae","Type":"ContainerDied","Data":"d8282dc7638cfdb8e5fde5964369d8c510ad04bfc16aacce2a36b11f48daab4b"} Nov 02 23:45:22.664350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.663975 5672 scope.go:122] "RemoveContainer" containerID="3b9b9942c195399706bbbf90e35f590942b5173db1d810c1b392ba06c235633c" Nov 02 23:45:22.665871 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.665840 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" Nov 02 23:45:22.666019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.665891 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn" event={"ID":"f7a89c90-63d6-44ab-a0a5-159fa8096ca5","Type":"ContainerDied","Data":"e4d542c3ca67041f8c60fd0db4fc38b25916318c30e1922138caf691826618bc"} Nov 02 23:45:22.676889 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.676867 5672 scope.go:122] "RemoveContainer" containerID="65b181d26f8462769e265a68a7521171821777d12b1e46452ec745d1160a93c9" Nov 02 23:45:22.713170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.713127 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" path="/var/lib/kubelet/pods/6d12bc18-bd4d-495b-8222-f9682c97db74/volumes" Nov 02 23:45:22.713588 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.713572 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" path="/var/lib/kubelet/pods/90894c0e-93e0-4e47-b550-80b4b66f091e/volumes" Nov 02 23:45:22.713938 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.713926 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" path="/var/lib/kubelet/pods/d4ed4836-7860-47fc-be7e-df5a7735b640/volumes" Nov 02 23:45:22.819130 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.819100 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc"] Nov 02 23:45:22.825191 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.825166 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894385-cf5nc"] Nov 02 23:45:22.927984 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.927907 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn"] Nov 02 23:45:22.933807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:22.933782 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894385-24kqn"] Nov 02 23:45:24.712564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:24.712530 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" path="/var/lib/kubelet/pods/517d111c-1dd2-4e24-99ab-ac1556f891ae/volumes" Nov 02 23:45:24.712965 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:24.712947 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" path="/var/lib/kubelet/pods/f7a89c90-63d6-44ab-a0a5-159fa8096ca5/volumes" Nov 02 23:45:26.020315 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:26.020280 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:45:39.943623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:39.943587 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb"] Nov 02 23:45:39.944197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:39.943829 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerName="istio-proxy" containerID="cri-o://ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7" gracePeriod=30 Nov 02 23:45:40.103178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:40.103126 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk"] Nov 02 23:45:40.103417 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:40.103393 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" containerID="cri-o://40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a" gracePeriod=30 Nov 02 23:45:42.686957 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:42.686921 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 23:45:42.687481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:42.686975 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 23:45:45.234124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.234096 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:45:45.247905 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.247875 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 23:45:45.248057 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.247945 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 23:45:45.305762 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.305731 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-kube-api-access-5tmpn\" (UniqueName: \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-kube-api-access-5tmpn\") pod \"939a37d5-5089-415f-be4c-f5ba19b0de18\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " Nov 02 23:45:45.305929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.305777 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-socket\") pod \"939a37d5-5089-415f-be4c-f5ba19b0de18\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " Nov 02 23:45:45.305929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.305832 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-certs\") pod \"939a37d5-5089-415f-be4c-f5ba19b0de18\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " Nov 02 23:45:45.305929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.305878 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-data\") pod \"939a37d5-5089-415f-be4c-f5ba19b0de18\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " Nov 02 23:45:45.305929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.305919 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939a37d5-5089-415f-be4c-f5ba19b0de18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939a37d5-5089-415f-be4c-f5ba19b0de18-istiod-ca-cert\") pod \"939a37d5-5089-415f-be4c-f5ba19b0de18\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " Nov 02 23:45:45.306202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.305962 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-envoy\") pod \"939a37d5-5089-415f-be4c-f5ba19b0de18\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " Nov 02 23:45:45.306202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.305998 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-podinfo\") pod \"939a37d5-5089-415f-be4c-f5ba19b0de18\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " Nov 02 23:45:45.306202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.306082 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-credential-socket\") pod \"939a37d5-5089-415f-be4c-f5ba19b0de18\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " Nov 02 23:45:45.306202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.306139 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-socket" pod "939a37d5-5089-415f-be4c-f5ba19b0de18" (UID: "939a37d5-5089-415f-be4c-f5ba19b0de18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:45.306202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.306168 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-token\" (UniqueName: \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-token\") pod \"939a37d5-5089-415f-be4c-f5ba19b0de18\" (UID: \"939a37d5-5089-415f-be4c-f5ba19b0de18\") " Nov 02 23:45:45.306507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.306356 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-data" pod "939a37d5-5089-415f-be4c-f5ba19b0de18" (UID: "939a37d5-5089-415f-be4c-f5ba19b0de18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:45.306507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.306474 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939a37d5-5089-415f-be4c-f5ba19b0de18-istiod-ca-cert" pod "939a37d5-5089-415f-be4c-f5ba19b0de18" (UID: "939a37d5-5089-415f-be4c-f5ba19b0de18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:45.306933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.306905 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.307062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.306938 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.307062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.306955 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939a37d5-5089-415f-be4c-f5ba19b0de18-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.307236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.307188 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-certs" pod "939a37d5-5089-415f-be4c-f5ba19b0de18" (UID: "939a37d5-5089-415f-be4c-f5ba19b0de18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:45.307303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.307270 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-credential-socket" pod "939a37d5-5089-415f-be4c-f5ba19b0de18" (UID: "939a37d5-5089-415f-be4c-f5ba19b0de18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:45.308346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.308318 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-podinfo" pod "939a37d5-5089-415f-be4c-f5ba19b0de18" (UID: "939a37d5-5089-415f-be4c-f5ba19b0de18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:45:45.308498 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.308473 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-kube-api-access-5tmpn" pod "939a37d5-5089-415f-be4c-f5ba19b0de18" (UID: "939a37d5-5089-415f-be4c-f5ba19b0de18"). InnerVolumeSpecName "kube-api-access-5tmpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:45.308919 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.308898 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-token" pod "939a37d5-5089-415f-be4c-f5ba19b0de18" (UID: "939a37d5-5089-415f-be4c-f5ba19b0de18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:45.309035 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.309017 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-envoy" pod "939a37d5-5089-415f-be4c-f5ba19b0de18" (UID: "939a37d5-5089-415f-be4c-f5ba19b0de18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:45.358060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.358029 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:45:45.407898 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.407870 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-podinfo\") pod \"fc8604f1-7218-42b8-9dff-499f12b878c2\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " Nov 02 23:45:45.408078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.407912 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-socket\") pod \"fc8604f1-7218-42b8-9dff-499f12b878c2\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " Nov 02 23:45:45.408078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.407944 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8604f1-7218-42b8-9dff-499f12b878c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8604f1-7218-42b8-9dff-499f12b878c2-istiod-ca-cert\") pod \"fc8604f1-7218-42b8-9dff-499f12b878c2\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " Nov 02 23:45:45.408078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.407972 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-credential-socket\") pod \"fc8604f1-7218-42b8-9dff-499f12b878c2\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " Nov 02 23:45:45.408078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408059 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-certs\") pod \"fc8604f1-7218-42b8-9dff-499f12b878c2\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " Nov 02 23:45:45.408383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408092 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-envoy\") pod \"fc8604f1-7218-42b8-9dff-499f12b878c2\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " Nov 02 23:45:45.408383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408124 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-kube-api-access-p2spz\" (UniqueName: \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-kube-api-access-p2spz\") pod \"fc8604f1-7218-42b8-9dff-499f12b878c2\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " Nov 02 23:45:45.408383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408144 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-token\" (UniqueName: \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-token\") pod \"fc8604f1-7218-42b8-9dff-499f12b878c2\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " Nov 02 23:45:45.408383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408182 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-data\") pod \"fc8604f1-7218-42b8-9dff-499f12b878c2\" (UID: \"fc8604f1-7218-42b8-9dff-499f12b878c2\") " Nov 02 23:45:45.408633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408612 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.408725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408639 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.408725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408656 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.408725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408670 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939a37d5-5089-415f-be4c-f5ba19b0de18-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.408725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408684 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.408725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408698 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmpn\" (UniqueName: \"kubernetes.io/projected/939a37d5-5089-415f-be4c-f5ba19b0de18-kube-api-access-5tmpn\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.408725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408695 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8604f1-7218-42b8-9dff-499f12b878c2-istiod-ca-cert" pod "fc8604f1-7218-42b8-9dff-499f12b878c2" (UID: "fc8604f1-7218-42b8-9dff-499f12b878c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:45:45.409202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408886 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-data" pod "fc8604f1-7218-42b8-9dff-499f12b878c2" (UID: "fc8604f1-7218-42b8-9dff-499f12b878c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:45.409202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.408977 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-socket" pod "fc8604f1-7218-42b8-9dff-499f12b878c2" (UID: "fc8604f1-7218-42b8-9dff-499f12b878c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:45.410300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.409285 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-certs" pod "fc8604f1-7218-42b8-9dff-499f12b878c2" (UID: "fc8604f1-7218-42b8-9dff-499f12b878c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:45.410300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.409986 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-credential-socket" pod "fc8604f1-7218-42b8-9dff-499f12b878c2" (UID: "fc8604f1-7218-42b8-9dff-499f12b878c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:45.411293 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.411266 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-podinfo" pod "fc8604f1-7218-42b8-9dff-499f12b878c2" (UID: "fc8604f1-7218-42b8-9dff-499f12b878c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:45:45.411399 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.411306 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-kube-api-access-p2spz" pod "fc8604f1-7218-42b8-9dff-499f12b878c2" (UID: "fc8604f1-7218-42b8-9dff-499f12b878c2"). InnerVolumeSpecName "kube-api-access-p2spz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:45.412426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.412407 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-envoy" pod "fc8604f1-7218-42b8-9dff-499f12b878c2" (UID: "fc8604f1-7218-42b8-9dff-499f12b878c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:45:45.412660 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.412641 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-token" pod "fc8604f1-7218-42b8-9dff-499f12b878c2" (UID: "fc8604f1-7218-42b8-9dff-499f12b878c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:45:45.510353 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.510316 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc8604f1-7218-42b8-9dff-499f12b878c2-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.510566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.510360 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.510566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.510376 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.510566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.510388 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.510566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.510400 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2spz\" (UniqueName: \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-kube-api-access-p2spz\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.510566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.510412 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.510566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.510425 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.510566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.510437 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc8604f1-7218-42b8-9dff-499f12b878c2-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.510566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.510449 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc8604f1-7218-42b8-9dff-499f12b878c2-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:45:45.817948 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.817866 5672 generic.go:356] "Generic (PLEG): container finished" podID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerID="40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a" exitCode=0 Nov 02 23:45:45.817948 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.817917 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" Nov 02 23:45:45.818233 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.817957 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" event={"ID":"fc8604f1-7218-42b8-9dff-499f12b878c2","Type":"ContainerDied","Data":"40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a"} Nov 02 23:45:45.818233 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.818027 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk" event={"ID":"fc8604f1-7218-42b8-9dff-499f12b878c2","Type":"ContainerDied","Data":"c4e4d6a682bba053e43ffe9d6daa849402274b6700a76821209511dfb17cb0d8"} Nov 02 23:45:45.818233 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.818057 5672 scope.go:122] "RemoveContainer" containerID="40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a" Nov 02 23:45:45.819359 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.819331 5672 generic.go:356] "Generic (PLEG): container finished" podID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerID="ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7" exitCode=0 Nov 02 23:45:45.819493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.819386 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" Nov 02 23:45:45.819493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.819413 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" event={"ID":"939a37d5-5089-415f-be4c-f5ba19b0de18","Type":"ContainerDied","Data":"ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7"} Nov 02 23:45:45.819493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.819475 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb" event={"ID":"939a37d5-5089-415f-be4c-f5ba19b0de18","Type":"ContainerDied","Data":"c39fbc9b73392848ab1b023641a3d560dfd0bb1d3c81dbf1184a3dba820b8cd9"} Nov 02 23:45:45.836670 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.836643 5672 scope.go:122] "RemoveContainer" containerID="40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a" Nov 02 23:45:45.837408 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:45.837363 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a\": container with ID starting with 40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a not found: ID does not exist" containerID="40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a" Nov 02 23:45:45.837525 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.837404 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a"} err="failed to get container status \"40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a\": rpc error: code = NotFound desc = could not find container \"40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a\": container with ID starting with 40c06b8c1be77a9fa70c3bb18c81091623e8654c6b76dd44daecdb742a192e5a not found: ID does not exist" Nov 02 23:45:45.837525 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.837433 5672 scope.go:122] "RemoveContainer" containerID="ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7" Nov 02 23:45:45.850001 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.849976 5672 scope.go:122] "RemoveContainer" containerID="ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7" Nov 02 23:45:45.850382 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:45.850360 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7\": container with ID starting with ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7 not found: ID does not exist" containerID="ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7" Nov 02 23:45:45.850464 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.850390 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7"} err="failed to get container status \"ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7\": rpc error: code = NotFound desc = could not find container \"ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7\": container with ID starting with ea7e3f006decf222a506c612d83404651e9bee1d7aa589c89738ab6aace965c7 not found: ID does not exist" Nov 02 23:45:45.971776 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:45.971735 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk"] Nov 02 23:45:46.017884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:46.017847 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-vzlmk"] Nov 02 23:45:46.119785 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:46.119702 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb"] Nov 02 23:45:46.123411 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:46.123380 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-6m4zb"] Nov 02 23:45:46.716319 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:46.716273 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" path="/var/lib/kubelet/pods/939a37d5-5089-415f-be4c-f5ba19b0de18/volumes" Nov 02 23:45:46.717056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:46.717032 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" path="/var/lib/kubelet/pods/fc8604f1-7218-42b8-9dff-499f12b878c2/volumes" Nov 02 23:45:49.608279 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:49.608238 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:45:53.634536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.634496 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n"] Nov 02 23:45:53.636168 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.636122 5672 cpu_manager.go:414] "containerMap: removing container" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" Nov 02 23:45:53.636356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.636340 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" Nov 02 23:45:53.636458 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.636448 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerName="cleanup" Nov 02 23:45:53.636532 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.636523 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerName="cleanup" Nov 02 23:45:53.636607 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.636598 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerName="cleanup" Nov 02 23:45:53.636680 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.636670 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerName="cleanup" Nov 02 23:45:53.636768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.636757 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerName="cleanup" Nov 02 23:45:53.636838 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.636828 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerName="cleanup" Nov 02 23:45:53.636911 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.636902 5672 cpu_manager.go:414] "containerMap: removing container" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerName="cleanup" Nov 02 23:45:53.636982 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.636972 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerName="cleanup" Nov 02 23:45:53.637051 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637042 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerName="cleanup" Nov 02 23:45:53.637128 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637119 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerName="cleanup" Nov 02 23:45:53.637229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637219 5672 cpu_manager.go:414] "containerMap: removing container" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerName="cleanup" Nov 02 23:45:53.637303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637295 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerName="cleanup" Nov 02 23:45:53.637383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637374 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" containerName="cleanup" Nov 02 23:45:53.637451 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637442 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" containerName="cleanup" Nov 02 23:45:53.637517 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637509 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" containerName="cleanup" Nov 02 23:45:53.637587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637578 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" containerName="cleanup" Nov 02 23:45:53.637677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637668 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerName="cleanup" Nov 02 23:45:53.637749 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637740 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerName="cleanup" Nov 02 23:45:53.637820 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637809 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerName="cleanup" Nov 02 23:45:53.637889 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637879 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerName="cleanup" Nov 02 23:45:53.637973 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.637963 5672 cpu_manager.go:414] "containerMap: removing container" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerName="cleanup" Nov 02 23:45:53.638043 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638034 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerName="cleanup" Nov 02 23:45:53.638136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638115 5672 cpu_manager.go:414] "containerMap: removing container" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerName="cleanup" Nov 02 23:45:53.638136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638130 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerName="cleanup" Nov 02 23:45:53.638136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638170 5672 cpu_manager.go:414] "containerMap: removing container" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerName="istio-proxy" Nov 02 23:45:53.638365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638176 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerName="istio-proxy" Nov 02 23:45:53.638365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638232 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerName="cleanup" Nov 02 23:45:53.638365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638268 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerName="cleanup" Nov 02 23:45:53.638365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638296 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerName="cleanup" Nov 02 23:45:53.638495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638443 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerName="cleanup" Nov 02 23:45:53.638495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638456 5672 cpu_manager.go:414] "containerMap: removing container" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerName="cleanup" Nov 02 23:45:53.638495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638461 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerName="cleanup" Nov 02 23:45:53.638495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638473 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerName="cleanup" Nov 02 23:45:53.638495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638478 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerName="cleanup" Nov 02 23:45:53.638495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638484 5672 cpu_manager.go:414] "containerMap: removing container" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerName="cleanup" Nov 02 23:45:53.638495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638488 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerName="cleanup" Nov 02 23:45:53.638495 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638497 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638502 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638508 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638513 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638521 5672 cpu_manager.go:414] "containerMap: removing container" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638525 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638542 5672 cpu_manager.go:414] "containerMap: removing container" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638546 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638557 5672 cpu_manager.go:414] "containerMap: removing container" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638561 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638567 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638571 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638667 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638675 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638681 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638687 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638693 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638699 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638705 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8604f1-7218-42b8-9dff-499f12b878c2" containerName="istio-proxy" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638711 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638716 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638722 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638728 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638732 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638738 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638744 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638749 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" containerName="cleanup" Nov 02 23:45:53.638805 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638755 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerName="cleanup" Nov 02 23:45:53.639644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638759 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerName="cleanup" Nov 02 23:45:53.639644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638900 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" containerName="cleanup" Nov 02 23:45:53.639644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638906 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="939a37d5-5089-415f-be4c-f5ba19b0de18" containerName="istio-proxy" Nov 02 23:45:53.639644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638912 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f89391-7aab-40c8-95d8-ae30eb9b2f50" containerName="cleanup" Nov 02 23:45:53.639644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638918 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerName="cleanup" Nov 02 23:45:53.639644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638923 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerName="cleanup" Nov 02 23:45:53.639644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638928 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerName="cleanup" Nov 02 23:45:53.639644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638934 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerName="cleanup" Nov 02 23:45:53.639644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.638939 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerName="cleanup" Nov 02 23:45:53.643624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.643602 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.688195 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.688141 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n"] Nov 02 23:45:53.703633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.703603 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxwh\" (UniqueName: \"kubernetes.io/projected/b2c8af1e-dd00-462a-b11a-9424a29321a9-kube-api-access-8fxwh\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.703808 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.703642 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-model-cache\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.703884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.703822 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-tmp-dir\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.703884 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.703873 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c8af1e-dd00-462a-b11a-9424a29321a9-tls-certs\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.703963 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.703942 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-home\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.704058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.703991 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-dshm\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.731359 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.731328 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-52dd59\"/\"kube-root-ca.crt\"" Nov 02 23:45:53.738317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.738293 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-52dd59\"/\"default-dockercfg-w2cfs\"" Nov 02 23:45:53.738317 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.738304 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-52dd59\"/\"explicit-model-52dd59-kserve-self-signed-certs\"" Nov 02 23:45:53.792332 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.792303 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-52dd59\"/\"openshift-service-ca.crt\"" Nov 02 23:45:53.805132 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.805098 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-tmp-dir\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.805380 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.805302 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c8af1e-dd00-462a-b11a-9424a29321a9-tls-certs\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.805475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.805387 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-home\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.805475 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.805463 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-dshm\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.805601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.805541 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxwh\" (UniqueName: \"kubernetes.io/projected/b2c8af1e-dd00-462a-b11a-9424a29321a9-kube-api-access-8fxwh\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.805601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.805592 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-model-cache\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.805720 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.805646 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-tmp-dir\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.806125 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.806098 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-model-cache\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.806263 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.806182 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-home\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.807886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.807869 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-dshm\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.808426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.808403 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c8af1e-dd00-462a-b11a-9424a29321a9-tls-certs\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.860387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.860322 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxwh\" (UniqueName: \"kubernetes.io/projected/b2c8af1e-dd00-462a-b11a-9424a29321a9-kube-api-access-8fxwh\") pod \"explicit-model-52dd59-kserve-9f6c868c7-dpd6n\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:53.959412 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:53.959314 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:54.330275 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:54.330249 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n"] Nov 02 23:45:54.332740 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:45:54.332708 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2c8af1e_dd00_462a_b11a_9424a29321a9.slice/crio-63c1856db9a63367577411c94fecb386c0eef6cb99b48e9fcd275b3f23907e26 WatchSource:0}: Error finding container 63c1856db9a63367577411c94fecb386c0eef6cb99b48e9fcd275b3f23907e26: Status 404 returned error can't find the container with id 63c1856db9a63367577411c94fecb386c0eef6cb99b48e9fcd275b3f23907e26 Nov 02 23:45:54.883759 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:54.883711 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" event={"ID":"b2c8af1e-dd00-462a-b11a-9424a29321a9","Type":"ContainerStarted","Data":"612d3a1dd0bcbdef2d248d1c57e5aae6e81f9b15734c2b6cc05d83f5b82b049b"} Nov 02 23:45:54.883759 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:54.883763 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" event={"ID":"b2c8af1e-dd00-462a-b11a-9424a29321a9","Type":"ContainerStarted","Data":"63c1856db9a63367577411c94fecb386c0eef6cb99b48e9fcd275b3f23907e26"} Nov 02 23:45:54.960376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:54.960338 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:45:54.992716 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:45:54.992656 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" podStartSLOduration=1.992639868 podStartE2EDuration="1.992639868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:54.991755013 +0000 UTC m=+3148.399066845" watchObservedRunningTime="2026-11-02 23:45:54.992639868 +0000 UTC m=+3148.399951700" Nov 02 23:45:56.023286 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:45:56.023249 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:46:03.960103 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:03.960074 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:46:03.973085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:03.973058 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:46:04.962684 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:04.962652 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:46:26.020671 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:46:26.020585 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:46:36.973434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:36.973398 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n"] Nov 02 23:46:46.974290 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:46.974256 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:46:51.993975 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:51.993934 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" podUID="b2c8af1e-dd00-462a-b11a-9424a29321a9" containerName="main" containerID="cri-o://612d3a1dd0bcbdef2d248d1c57e5aae6e81f9b15734c2b6cc05d83f5b82b049b" gracePeriod=45 Nov 02 23:46:52.263192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.263140 5672 generic.go:356] "Generic (PLEG): container finished" podID="b2c8af1e-dd00-462a-b11a-9424a29321a9" containerID="612d3a1dd0bcbdef2d248d1c57e5aae6e81f9b15734c2b6cc05d83f5b82b049b" exitCode=0 Nov 02 23:46:52.263343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.263190 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" event={"ID":"b2c8af1e-dd00-462a-b11a-9424a29321a9","Type":"ContainerDied","Data":"612d3a1dd0bcbdef2d248d1c57e5aae6e81f9b15734c2b6cc05d83f5b82b049b"} Nov 02 23:46:52.263343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.263254 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" event={"ID":"b2c8af1e-dd00-462a-b11a-9424a29321a9","Type":"ContainerDied","Data":"63c1856db9a63367577411c94fecb386c0eef6cb99b48e9fcd275b3f23907e26"} Nov 02 23:46:52.263343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.263271 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63c1856db9a63367577411c94fecb386c0eef6cb99b48e9fcd275b3f23907e26" Nov 02 23:46:52.273816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.273797 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:46:52.381275 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.381240 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-model-cache\") pod \"b2c8af1e-dd00-462a-b11a-9424a29321a9\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " Nov 02 23:46:52.381463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.381301 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c8af1e-dd00-462a-b11a-9424a29321a9-kube-api-access-8fxwh\" (UniqueName: \"kubernetes.io/projected/b2c8af1e-dd00-462a-b11a-9424a29321a9-kube-api-access-8fxwh\") pod \"b2c8af1e-dd00-462a-b11a-9424a29321a9\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " Nov 02 23:46:52.381463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.381339 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-dshm\") pod \"b2c8af1e-dd00-462a-b11a-9424a29321a9\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " Nov 02 23:46:52.381463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.381380 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-tmp-dir\") pod \"b2c8af1e-dd00-462a-b11a-9424a29321a9\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " Nov 02 23:46:52.381637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.381554 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-model-cache" pod "b2c8af1e-dd00-462a-b11a-9424a29321a9" (UID: "b2c8af1e-dd00-462a-b11a-9424a29321a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:52.381637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.381590 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c8af1e-dd00-462a-b11a-9424a29321a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c8af1e-dd00-462a-b11a-9424a29321a9-tls-certs\") pod \"b2c8af1e-dd00-462a-b11a-9424a29321a9\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " Nov 02 23:46:52.381637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.381631 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-home\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-home\") pod \"b2c8af1e-dd00-462a-b11a-9424a29321a9\" (UID: \"b2c8af1e-dd00-462a-b11a-9424a29321a9\") " Nov 02 23:46:52.381807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.381639 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-tmp-dir" pod "b2c8af1e-dd00-462a-b11a-9424a29321a9" (UID: "b2c8af1e-dd00-462a-b11a-9424a29321a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:52.381900 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.381883 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-home" pod "b2c8af1e-dd00-462a-b11a-9424a29321a9" (UID: "b2c8af1e-dd00-462a-b11a-9424a29321a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:52.382114 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.382098 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:46:52.382295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.382116 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:46:52.382295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.382127 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:46:52.384218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.384186 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-dshm" pod "b2c8af1e-dd00-462a-b11a-9424a29321a9" (UID: "b2c8af1e-dd00-462a-b11a-9424a29321a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:46:52.384334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.384214 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c8af1e-dd00-462a-b11a-9424a29321a9-tls-certs" pod "b2c8af1e-dd00-462a-b11a-9424a29321a9" (UID: "b2c8af1e-dd00-462a-b11a-9424a29321a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:46:52.384405 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.384330 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c8af1e-dd00-462a-b11a-9424a29321a9-kube-api-access-8fxwh" pod "b2c8af1e-dd00-462a-b11a-9424a29321a9" (UID: "b2c8af1e-dd00-462a-b11a-9424a29321a9"). InnerVolumeSpecName "kube-api-access-8fxwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:46:52.483804 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.483773 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c8af1e-dd00-462a-b11a-9424a29321a9-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:46:52.483804 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.483802 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxwh\" (UniqueName: \"kubernetes.io/projected/b2c8af1e-dd00-462a-b11a-9424a29321a9-kube-api-access-8fxwh\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:46:52.484007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:52.483812 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c8af1e-dd00-462a-b11a-9424a29321a9-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:46:53.270164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:53.270127 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n" Nov 02 23:46:53.425400 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:53.425348 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n"] Nov 02 23:46:53.427773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:53.427746 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-52dd59/explicit-model-52dd59-kserve-9f6c868c7-dpd6n"] Nov 02 23:46:54.722981 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:46:54.722939 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c8af1e-dd00-462a-b11a-9424a29321a9" path="/var/lib/kubelet/pods/b2c8af1e-dd00-462a-b11a-9424a29321a9/volumes" Nov 02 23:46:56.021008 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:46:56.020969 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:47:14.429441 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:14.429409 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g"] Nov 02 23:47:19.430605 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:19.430555 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" podUID="957cd081-e9dc-42ea-8376-551baa2df4be" containerName="payload-processing" containerID="cri-o://1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a" gracePeriod=25 Nov 02 23:47:19.696924 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:19.696866 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:47:19.791944 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:19.791908 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957cd081-e9dc-42ea-8376-551baa2df4be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957cd081-e9dc-42ea-8376-551baa2df4be-plugins-config-volume\") pod \"957cd081-e9dc-42ea-8376-551baa2df4be\" (UID: \"957cd081-e9dc-42ea-8376-551baa2df4be\") " Nov 02 23:47:19.792133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:19.792117 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957cd081-e9dc-42ea-8376-551baa2df4be-kube-api-access-qd64x\" (UniqueName: \"kubernetes.io/projected/957cd081-e9dc-42ea-8376-551baa2df4be-kube-api-access-qd64x\") pod \"957cd081-e9dc-42ea-8376-551baa2df4be\" (UID: \"957cd081-e9dc-42ea-8376-551baa2df4be\") " Nov 02 23:47:19.792418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:19.792390 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957cd081-e9dc-42ea-8376-551baa2df4be-plugins-config-volume" pod "957cd081-e9dc-42ea-8376-551baa2df4be" (UID: "957cd081-e9dc-42ea-8376-551baa2df4be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:47:19.792567 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:19.792550 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/957cd081-e9dc-42ea-8376-551baa2df4be-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:19.794217 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:19.794195 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957cd081-e9dc-42ea-8376-551baa2df4be-kube-api-access-qd64x" pod "957cd081-e9dc-42ea-8376-551baa2df4be" (UID: "957cd081-e9dc-42ea-8376-551baa2df4be"). InnerVolumeSpecName "kube-api-access-qd64x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:47:19.893438 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:19.893404 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd64x\" (UniqueName: \"kubernetes.io/projected/957cd081-e9dc-42ea-8376-551baa2df4be-kube-api-access-qd64x\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:20.461263 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:20.461224 5672 generic.go:356] "Generic (PLEG): container finished" podID="957cd081-e9dc-42ea-8376-551baa2df4be" containerID="1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a" exitCode=0 Nov 02 23:47:20.461774 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:20.461280 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" Nov 02 23:47:20.461774 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:20.461296 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" event={"ID":"957cd081-e9dc-42ea-8376-551baa2df4be","Type":"ContainerDied","Data":"1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a"} Nov 02 23:47:20.461774 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:20.461347 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g" event={"ID":"957cd081-e9dc-42ea-8376-551baa2df4be","Type":"ContainerDied","Data":"536bd8c17e2d9959505c4ed2c47b1a6c0ffe6e98b3aeec971fd040d37f81eabe"} Nov 02 23:47:20.461774 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:20.461375 5672 scope.go:122] "RemoveContainer" containerID="1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a" Nov 02 23:47:20.484590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:20.484568 5672 scope.go:122] "RemoveContainer" containerID="1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a" Nov 02 23:47:20.485117 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:47:20.485092 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a\": container with ID starting with 1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a not found: ID does not exist" containerID="1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a" Nov 02 23:47:20.485251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:20.485124 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a"} err="failed to get container status \"1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a\": rpc error: code = NotFound desc = could not find container \"1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a\": container with ID starting with 1d6301e083fed5e4a602539da191e562ecf0f6a7389891511c8d245e7214827a not found: ID does not exist" Nov 02 23:47:20.614668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:20.614631 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g"] Nov 02 23:47:20.618057 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:20.618029 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52dd59-7df5f78679-8md5g"] Nov 02 23:47:20.712917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:20.712829 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957cd081-e9dc-42ea-8376-551baa2df4be" path="/var/lib/kubelet/pods/957cd081-e9dc-42ea-8376-551baa2df4be/volumes" Nov 02 23:47:26.020941 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:47:26.020906 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:47:34.003513 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:34.003477 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:47:40.853055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853021 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c"] Nov 02 23:47:40.853550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853523 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerName="cleanup" Nov 02 23:47:40.853550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853538 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a4d4c2-a470-4945-85a4-2ceec73fd11b" containerName="cleanup" Nov 02 23:47:40.853550 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853551 5672 cpu_manager.go:414] "containerMap: removing container" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853556 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853563 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853568 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853585 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853590 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd99463-1bb3-4b55-a805-49dcae325120" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853603 5672 cpu_manager.go:414] "containerMap: removing container" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853608 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853616 5672 cpu_manager.go:414] "containerMap: removing container" podUID="b2c8af1e-dd00-462a-b11a-9424a29321a9" containerName="main" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853621 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c8af1e-dd00-462a-b11a-9424a29321a9" containerName="main" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853632 5672 cpu_manager.go:414] "containerMap: removing container" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853636 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853645 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853649 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerName="cleanup" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853657 5672 cpu_manager.go:414] "containerMap: removing container" podUID="957cd081-e9dc-42ea-8376-551baa2df4be" containerName="payload-processing" Nov 02 23:47:40.853658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853661 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957cd081-e9dc-42ea-8376-551baa2df4be" containerName="payload-processing" Nov 02 23:47:40.854126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853743 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c8af1e-dd00-462a-b11a-9424a29321a9" containerName="main" Nov 02 23:47:40.854126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853753 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="558a2c75-72b2-41cf-b3e4-9d2dc53437f3" containerName="cleanup" Nov 02 23:47:40.854126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853759 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cef0b79-c23e-45ff-8936-92f4f0ea3708" containerName="cleanup" Nov 02 23:47:40.854126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853764 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="517d111c-1dd2-4e24-99ab-ac1556f891ae" containerName="cleanup" Nov 02 23:47:40.854126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853770 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="957cd081-e9dc-42ea-8376-551baa2df4be" containerName="payload-processing" Nov 02 23:47:40.854126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853776 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d12bc18-bd4d-495b-8222-f9682c97db74" containerName="cleanup" Nov 02 23:47:40.854126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853783 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a89c90-63d6-44ab-a0a5-159fa8096ca5" containerName="cleanup" Nov 02 23:47:40.854126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853790 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="90894c0e-93e0-4e47-b550-80b4b66f091e" containerName="cleanup" Nov 02 23:47:40.854126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.853795 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ed4836-7860-47fc-be7e-df5a7735b640" containerName="cleanup" Nov 02 23:47:40.858295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.858275 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:40.909871 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.909843 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c"] Nov 02 23:47:40.930318 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.930286 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlzsn\" (UniqueName: \"kubernetes.io/projected/f6be90a6-8029-446b-a4dd-94080cfccb29-kube-api-access-rlzsn\") pod \"maas-api-revoke-sub-005fbc5e-zjw5c\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:40.930465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.930410 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-005fbc5e-zjw5c\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:40.930526 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.930512 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-005fbc5e-zjw5c\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:40.952602 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.952572 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:47:40.952602 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.952591 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:47:40.953281 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.953261 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:47:40.953356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:40.953335 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:47:41.031162 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:41.031128 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlzsn\" (UniqueName: \"kubernetes.io/projected/f6be90a6-8029-446b-a4dd-94080cfccb29-kube-api-access-rlzsn\") pod \"maas-api-revoke-sub-005fbc5e-zjw5c\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:41.031336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:41.031190 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-005fbc5e-zjw5c\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:41.031336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:41.031257 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-005fbc5e-zjw5c\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:41.032119 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:41.032088 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-005fbc5e-zjw5c\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:41.033807 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:41.033786 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-005fbc5e-zjw5c\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:41.084273 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:41.084248 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlzsn\" (UniqueName: \"kubernetes.io/projected/f6be90a6-8029-446b-a4dd-94080cfccb29-kube-api-access-rlzsn\") pod \"maas-api-revoke-sub-005fbc5e-zjw5c\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:41.173341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:41.173270 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:41.540947 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:41.540917 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c"] Nov 02 23:47:41.543322 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:47:41.543290 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6be90a6_8029_446b_a4dd_94080cfccb29.slice/crio-3cd78703cb7c48cb281c591e34cfa5ae5b42027331ab1cbddef5d98405393c19 WatchSource:0}: Error finding container 3cd78703cb7c48cb281c591e34cfa5ae5b42027331ab1cbddef5d98405393c19: Status 404 returned error can't find the container with id 3cd78703cb7c48cb281c591e34cfa5ae5b42027331ab1cbddef5d98405393c19 Nov 02 23:47:41.599076 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:41.599050 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" event={"ID":"f6be90a6-8029-446b-a4dd-94080cfccb29","Type":"ContainerStarted","Data":"3cd78703cb7c48cb281c591e34cfa5ae5b42027331ab1cbddef5d98405393c19"} Nov 02 23:47:42.607240 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:42.607203 5672 generic.go:356] "Generic (PLEG): container finished" podID="f6be90a6-8029-446b-a4dd-94080cfccb29" containerID="6a4f162be64f211f0d9c737f636ac6a2100528d073e780d1b6ff11d8c5b453e3" exitCode=0 Nov 02 23:47:42.607633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:42.607290 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" event={"ID":"f6be90a6-8029-446b-a4dd-94080cfccb29","Type":"ContainerDied","Data":"6a4f162be64f211f0d9c737f636ac6a2100528d073e780d1b6ff11d8c5b453e3"} Nov 02 23:47:43.765583 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:43.765558 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:43.871335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:43.871212 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-xks-ca\") pod \"f6be90a6-8029-446b-a4dd-94080cfccb29\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " Nov 02 23:47:43.871713 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:43.871683 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-openshift-ca\") pod \"f6be90a6-8029-446b-a4dd-94080cfccb29\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " Nov 02 23:47:43.872050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:43.872022 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-openshift-ca" pod "f6be90a6-8029-446b-a4dd-94080cfccb29" (UID: "f6be90a6-8029-446b-a4dd-94080cfccb29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:47:43.872197 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:43.872172 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6be90a6-8029-446b-a4dd-94080cfccb29-kube-api-access-rlzsn\" (UniqueName: \"kubernetes.io/projected/f6be90a6-8029-446b-a4dd-94080cfccb29-kube-api-access-rlzsn\") pod \"f6be90a6-8029-446b-a4dd-94080cfccb29\" (UID: \"f6be90a6-8029-446b-a4dd-94080cfccb29\") " Nov 02 23:47:43.872613 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:43.872595 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:43.875136 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:43.875110 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-xks-ca" pod "f6be90a6-8029-446b-a4dd-94080cfccb29" (UID: "f6be90a6-8029-446b-a4dd-94080cfccb29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:47:43.875511 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:43.875484 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6be90a6-8029-446b-a4dd-94080cfccb29-kube-api-access-rlzsn" pod "f6be90a6-8029-446b-a4dd-94080cfccb29" (UID: "f6be90a6-8029-446b-a4dd-94080cfccb29"). InnerVolumeSpecName "kube-api-access-rlzsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:47:43.973868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:43.973836 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlzsn\" (UniqueName: \"kubernetes.io/projected/f6be90a6-8029-446b-a4dd-94080cfccb29-kube-api-access-rlzsn\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:43.973868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:43.973862 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6be90a6-8029-446b-a4dd-94080cfccb29-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:44.620793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:44.620761 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" Nov 02 23:47:44.620989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:44.620792 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c" event={"ID":"f6be90a6-8029-446b-a4dd-94080cfccb29","Type":"ContainerDied","Data":"3cd78703cb7c48cb281c591e34cfa5ae5b42027331ab1cbddef5d98405393c19"} Nov 02 23:47:44.620989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:44.620853 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd78703cb7c48cb281c591e34cfa5ae5b42027331ab1cbddef5d98405393c19" Nov 02 23:47:51.565175 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.565130 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss"] Nov 02 23:47:51.565650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.565636 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f6be90a6-8029-446b-a4dd-94080cfccb29" containerName="revoke-keys" Nov 02 23:47:51.565703 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.565651 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6be90a6-8029-446b-a4dd-94080cfccb29" containerName="revoke-keys" Nov 02 23:47:51.565768 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.565759 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6be90a6-8029-446b-a4dd-94080cfccb29" containerName="revoke-keys" Nov 02 23:47:51.569292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.569267 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:51.618292 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.618240 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:47:51.656445 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.656408 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72800408-6af7-4109-8ddc-277f0511a589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7142b1e5-2dsss\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:51.656652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.656583 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72800408-6af7-4109-8ddc-277f0511a589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7142b1e5-2dsss\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:51.656703 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.656682 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26kz\" (UniqueName: \"kubernetes.io/projected/72800408-6af7-4109-8ddc-277f0511a589-kube-api-access-g26kz\") pod \"maas-api-revoke-sub-7142b1e5-2dsss\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:51.664668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.664643 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:47:51.664799 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.664736 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:47:51.664880 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.664862 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:47:51.667521 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.667499 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss"] Nov 02 23:47:51.757361 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.757323 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26kz\" (UniqueName: \"kubernetes.io/projected/72800408-6af7-4109-8ddc-277f0511a589-kube-api-access-g26kz\") pod \"maas-api-revoke-sub-7142b1e5-2dsss\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:51.757564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.757402 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72800408-6af7-4109-8ddc-277f0511a589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7142b1e5-2dsss\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:51.757564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.757516 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72800408-6af7-4109-8ddc-277f0511a589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7142b1e5-2dsss\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:51.758061 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.758034 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72800408-6af7-4109-8ddc-277f0511a589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7142b1e5-2dsss\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:51.759810 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.759786 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72800408-6af7-4109-8ddc-277f0511a589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7142b1e5-2dsss\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:51.810205 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.810140 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26kz\" (UniqueName: \"kubernetes.io/projected/72800408-6af7-4109-8ddc-277f0511a589-kube-api-access-g26kz\") pod \"maas-api-revoke-sub-7142b1e5-2dsss\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:51.882616 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:51.882526 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:52.250346 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:52.250312 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss"] Nov 02 23:47:52.252871 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:47:52.252835 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72800408_6af7_4109_8ddc_277f0511a589.slice/crio-b126f374e9cae87cdbcaed56725385390b15e0e1761e5a49483a17b0b9f3cd0b WatchSource:0}: Error finding container b126f374e9cae87cdbcaed56725385390b15e0e1761e5a49483a17b0b9f3cd0b: Status 404 returned error can't find the container with id b126f374e9cae87cdbcaed56725385390b15e0e1761e5a49483a17b0b9f3cd0b Nov 02 23:47:52.689572 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:52.689536 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" event={"ID":"72800408-6af7-4109-8ddc-277f0511a589","Type":"ContainerStarted","Data":"cd16b4c24e030b9cbaf6004f0b180b5f2449998c74d034bb32048666a1c50fad"} Nov 02 23:47:52.689978 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:52.689596 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" event={"ID":"72800408-6af7-4109-8ddc-277f0511a589","Type":"ContainerStarted","Data":"b126f374e9cae87cdbcaed56725385390b15e0e1761e5a49483a17b0b9f3cd0b"} Nov 02 23:47:52.803841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:52.803718 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" podStartSLOduration=1.803698083 podStartE2EDuration="1.803698083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:47:52.802782034 +0000 UTC m=+3266.210093864" watchObservedRunningTime="2026-11-02 23:47:52.803698083 +0000 UTC m=+3266.211009912" Nov 02 23:47:53.698300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:53.698263 5672 generic.go:356] "Generic (PLEG): container finished" podID="72800408-6af7-4109-8ddc-277f0511a589" containerID="cd16b4c24e030b9cbaf6004f0b180b5f2449998c74d034bb32048666a1c50fad" exitCode=0 Nov 02 23:47:53.698709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:53.698346 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" event={"ID":"72800408-6af7-4109-8ddc-277f0511a589","Type":"ContainerDied","Data":"cd16b4c24e030b9cbaf6004f0b180b5f2449998c74d034bb32048666a1c50fad"} Nov 02 23:47:54.842427 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:54.842395 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:55.005534 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.005503 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72800408-6af7-4109-8ddc-277f0511a589-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72800408-6af7-4109-8ddc-277f0511a589-maas-api-xks-ca\") pod \"72800408-6af7-4109-8ddc-277f0511a589\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " Nov 02 23:47:55.005721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.005604 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72800408-6af7-4109-8ddc-277f0511a589-kube-api-access-g26kz\" (UniqueName: \"kubernetes.io/projected/72800408-6af7-4109-8ddc-277f0511a589-kube-api-access-g26kz\") pod \"72800408-6af7-4109-8ddc-277f0511a589\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " Nov 02 23:47:55.005721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.005677 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72800408-6af7-4109-8ddc-277f0511a589-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72800408-6af7-4109-8ddc-277f0511a589-maas-api-openshift-ca\") pod \"72800408-6af7-4109-8ddc-277f0511a589\" (UID: \"72800408-6af7-4109-8ddc-277f0511a589\") " Nov 02 23:47:55.006034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.006006 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72800408-6af7-4109-8ddc-277f0511a589-maas-api-openshift-ca" pod "72800408-6af7-4109-8ddc-277f0511a589" (UID: "72800408-6af7-4109-8ddc-277f0511a589"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:47:55.007710 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.007681 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72800408-6af7-4109-8ddc-277f0511a589-maas-api-xks-ca" pod "72800408-6af7-4109-8ddc-277f0511a589" (UID: "72800408-6af7-4109-8ddc-277f0511a589"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:47:55.007802 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.007741 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72800408-6af7-4109-8ddc-277f0511a589-kube-api-access-g26kz" pod "72800408-6af7-4109-8ddc-277f0511a589" (UID: "72800408-6af7-4109-8ddc-277f0511a589"). InnerVolumeSpecName "kube-api-access-g26kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:47:55.107796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.107754 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72800408-6af7-4109-8ddc-277f0511a589-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:55.107796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.107799 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72800408-6af7-4109-8ddc-277f0511a589-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:55.108015 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.107816 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g26kz\" (UniqueName: \"kubernetes.io/projected/72800408-6af7-4109-8ddc-277f0511a589-kube-api-access-g26kz\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:47:55.714679 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.714649 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" Nov 02 23:47:55.714900 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.714683 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss" event={"ID":"72800408-6af7-4109-8ddc-277f0511a589","Type":"ContainerDied","Data":"b126f374e9cae87cdbcaed56725385390b15e0e1761e5a49483a17b0b9f3cd0b"} Nov 02 23:47:55.714900 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:47:55.714737 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b126f374e9cae87cdbcaed56725385390b15e0e1761e5a49483a17b0b9f3cd0b" Nov 02 23:47:56.020397 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:47:56.020358 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:48:15.241329 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.241280 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq"] Nov 02 23:48:15.241874 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.241813 5672 cpu_manager.go:414] "containerMap: removing container" podUID="72800408-6af7-4109-8ddc-277f0511a589" containerName="revoke-keys" Nov 02 23:48:15.241874 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.241826 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72800408-6af7-4109-8ddc-277f0511a589" containerName="revoke-keys" Nov 02 23:48:15.241973 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.241902 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="72800408-6af7-4109-8ddc-277f0511a589" containerName="revoke-keys" Nov 02 23:48:15.245270 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.245250 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.340358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.340324 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c343ed20-e745-4bf0-a1fa-359f982d147d-tls-certs\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.340358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.340364 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-tmp-dir\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.340575 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.340391 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-home\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.340575 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.340547 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-dshm\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.340662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.340642 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w9xn\" (UniqueName: \"kubernetes.io/projected/c343ed20-e745-4bf0-a1fa-359f982d147d-kube-api-access-7w9xn\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.340704 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.340687 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-model-cache\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.341313 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.341290 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-313523\"/\"kube-root-ca.crt\"" Nov 02 23:48:15.341375 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.341355 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-313523\"/\"openshift-service-ca.crt\"" Nov 02 23:48:15.341529 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.341513 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-313523\"/\"noten-model-313523-kserve-self-signed-certs\"" Nov 02 23:48:15.341660 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.341643 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-313523\"/\"default-dockercfg-blb58\"" Nov 02 23:48:15.344001 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.343985 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq"] Nov 02 23:48:15.442141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.442107 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-dshm\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.442330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.442193 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w9xn\" (UniqueName: \"kubernetes.io/projected/c343ed20-e745-4bf0-a1fa-359f982d147d-kube-api-access-7w9xn\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.442330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.442229 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-model-cache\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.442330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.442290 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c343ed20-e745-4bf0-a1fa-359f982d147d-tls-certs\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.442330 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.442310 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-tmp-dir\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.442556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.442373 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-home\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.442695 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.442672 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-tmp-dir\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.442785 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.442753 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-model-cache\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.442835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.442816 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-home\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.444707 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.444678 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c343ed20-e745-4bf0-a1fa-359f982d147d-tls-certs\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.444841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.444800 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-dshm\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.495393 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.495325 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w9xn\" (UniqueName: \"kubernetes.io/projected/c343ed20-e745-4bf0-a1fa-359f982d147d-kube-api-access-7w9xn\") pod \"noten-model-313523-kserve-f66665869-w46gq\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.558252 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.558219 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:15.943683 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:15.943655 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq"] Nov 02 23:48:15.945481 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:48:15.945453 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc343ed20_e745_4bf0_a1fa_359f982d147d.slice/crio-820988e2afdd26551642aa253a8fd8fe69b5e94ef33d93b4a014c5cd5c77ed48 WatchSource:0}: Error finding container 820988e2afdd26551642aa253a8fd8fe69b5e94ef33d93b4a014c5cd5c77ed48: Status 404 returned error can't find the container with id 820988e2afdd26551642aa253a8fd8fe69b5e94ef33d93b4a014c5cd5c77ed48 Nov 02 23:48:16.844637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:16.844604 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" event={"ID":"c343ed20-e745-4bf0-a1fa-359f982d147d","Type":"ContainerStarted","Data":"bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221"} Nov 02 23:48:16.845055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:16.844642 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" event={"ID":"c343ed20-e745-4bf0-a1fa-359f982d147d","Type":"ContainerStarted","Data":"820988e2afdd26551642aa253a8fd8fe69b5e94ef33d93b4a014c5cd5c77ed48"} Nov 02 23:48:16.954264 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:16.954192 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" podStartSLOduration=1.954170006 podStartE2EDuration="1.954170006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:48:16.950826613 +0000 UTC m=+3290.358138457" watchObservedRunningTime="2026-11-02 23:48:16.954170006 +0000 UTC m=+3290.361481837" Nov 02 23:48:17.559087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:17.559056 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:25.558819 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:25.558788 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:25.572865 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:25.572835 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:25.916564 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:25.916474 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:48:26.021077 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:48:26.021041 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:48:49.549195 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:49.549143 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:48:56.020495 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:48:56.020452 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:48:59.201684 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:48:59.201647 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq"] Nov 02 23:49:14.222416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.222316 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" podUID="c343ed20-e745-4bf0-a1fa-359f982d147d" containerName="main" containerID="cri-o://bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221" gracePeriod=45 Nov 02 23:49:14.481262 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.481198 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:49:14.509662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.509637 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c343ed20-e745-4bf0-a1fa-359f982d147d-tls-certs\" (UniqueName: \"kubernetes.io/secret/c343ed20-e745-4bf0-a1fa-359f982d147d-tls-certs\") pod \"c343ed20-e745-4bf0-a1fa-359f982d147d\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " Nov 02 23:49:14.509796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.509681 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-home\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-home\") pod \"c343ed20-e745-4bf0-a1fa-359f982d147d\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " Nov 02 23:49:14.509796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.509732 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-tmp-dir\") pod \"c343ed20-e745-4bf0-a1fa-359f982d147d\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " Nov 02 23:49:14.509796 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.509788 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-model-cache\") pod \"c343ed20-e745-4bf0-a1fa-359f982d147d\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " Nov 02 23:49:14.509989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.509857 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-dshm\") pod \"c343ed20-e745-4bf0-a1fa-359f982d147d\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " Nov 02 23:49:14.509989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.509923 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c343ed20-e745-4bf0-a1fa-359f982d147d-kube-api-access-7w9xn\" (UniqueName: \"kubernetes.io/projected/c343ed20-e745-4bf0-a1fa-359f982d147d-kube-api-access-7w9xn\") pod \"c343ed20-e745-4bf0-a1fa-359f982d147d\" (UID: \"c343ed20-e745-4bf0-a1fa-359f982d147d\") " Nov 02 23:49:14.510087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.510009 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-home" pod "c343ed20-e745-4bf0-a1fa-359f982d147d" (UID: "c343ed20-e745-4bf0-a1fa-359f982d147d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:49:14.510087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.510019 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-tmp-dir" pod "c343ed20-e745-4bf0-a1fa-359f982d147d" (UID: "c343ed20-e745-4bf0-a1fa-359f982d147d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:49:14.510087 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.510057 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-model-cache" pod "c343ed20-e745-4bf0-a1fa-359f982d147d" (UID: "c343ed20-e745-4bf0-a1fa-359f982d147d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:49:14.510440 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.510417 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:14.510440 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.510440 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:14.510643 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.510454 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:14.512038 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.512012 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c343ed20-e745-4bf0-a1fa-359f982d147d-tls-certs" pod "c343ed20-e745-4bf0-a1fa-359f982d147d" (UID: "c343ed20-e745-4bf0-a1fa-359f982d147d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:49:14.512176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.512046 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c343ed20-e745-4bf0-a1fa-359f982d147d-kube-api-access-7w9xn" pod "c343ed20-e745-4bf0-a1fa-359f982d147d" (UID: "c343ed20-e745-4bf0-a1fa-359f982d147d"). InnerVolumeSpecName "kube-api-access-7w9xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:49:14.512176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.512056 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-dshm" pod "c343ed20-e745-4bf0-a1fa-359f982d147d" (UID: "c343ed20-e745-4bf0-a1fa-359f982d147d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:49:14.611683 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.611647 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c343ed20-e745-4bf0-a1fa-359f982d147d-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:14.611683 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.611675 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w9xn\" (UniqueName: \"kubernetes.io/projected/c343ed20-e745-4bf0-a1fa-359f982d147d-kube-api-access-7w9xn\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:14.611683 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:14.611685 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c343ed20-e745-4bf0-a1fa-359f982d147d-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:15.231107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:15.231057 5672 generic.go:356] "Generic (PLEG): container finished" podID="c343ed20-e745-4bf0-a1fa-359f982d147d" containerID="bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221" exitCode=0 Nov 02 23:49:15.231107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:15.231099 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" Nov 02 23:49:15.231626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:15.231137 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" event={"ID":"c343ed20-e745-4bf0-a1fa-359f982d147d","Type":"ContainerDied","Data":"bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221"} Nov 02 23:49:15.231626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:15.231213 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq" event={"ID":"c343ed20-e745-4bf0-a1fa-359f982d147d","Type":"ContainerDied","Data":"820988e2afdd26551642aa253a8fd8fe69b5e94ef33d93b4a014c5cd5c77ed48"} Nov 02 23:49:15.231626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:15.231236 5672 scope.go:122] "RemoveContainer" containerID="bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221" Nov 02 23:49:15.241356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:15.241328 5672 scope.go:122] "RemoveContainer" containerID="bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221" Nov 02 23:49:15.241650 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:49:15.241628 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221\": container with ID starting with bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221 not found: ID does not exist" containerID="bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221" Nov 02 23:49:15.241715 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:15.241656 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221"} err="failed to get container status \"bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221\": rpc error: code = NotFound desc = could not find container \"bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221\": container with ID starting with bfcb836577c99950fcd0653a6dc67d3576fd7bd6737e2104705f06ea83d8b221 not found: ID does not exist" Nov 02 23:49:15.385360 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:15.385326 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq"] Nov 02 23:49:15.431616 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:15.431586 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-313523/noten-model-313523-kserve-f66665869-w46gq"] Nov 02 23:49:16.713818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:16.713781 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c343ed20-e745-4bf0-a1fa-359f982d147d" path="/var/lib/kubelet/pods/c343ed20-e745-4bf0-a1fa-359f982d147d/volumes" Nov 02 23:49:22.178860 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.178819 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b"] Nov 02 23:49:22.179571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.179549 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c343ed20-e745-4bf0-a1fa-359f982d147d" containerName="main" Nov 02 23:49:22.179672 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.179574 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c343ed20-e745-4bf0-a1fa-359f982d147d" containerName="main" Nov 02 23:49:22.179771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.179757 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c343ed20-e745-4bf0-a1fa-359f982d147d" containerName="main" Nov 02 23:49:22.184633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.184607 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:22.236165 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.236113 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b"] Nov 02 23:49:22.279079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.279042 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:49:22.279289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.279164 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:49:22.279289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.279196 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:49:22.279828 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.279811 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:49:22.296406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.296380 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03bc356b-c2t9b\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:22.296535 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.296421 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03bc356b-c2t9b\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:22.296638 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.296620 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm76f\" (UniqueName: \"kubernetes.io/projected/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-kube-api-access-mm76f\") pod \"maas-api-revoke-sub-03bc356b-c2t9b\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:22.397947 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.397911 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm76f\" (UniqueName: \"kubernetes.io/projected/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-kube-api-access-mm76f\") pod \"maas-api-revoke-sub-03bc356b-c2t9b\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:22.398118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.397960 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03bc356b-c2t9b\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:22.398118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.397991 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03bc356b-c2t9b\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:22.398800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.398780 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03bc356b-c2t9b\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:22.400505 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.400486 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03bc356b-c2t9b\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:22.452516 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.452447 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm76f\" (UniqueName: \"kubernetes.io/projected/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-kube-api-access-mm76f\") pod \"maas-api-revoke-sub-03bc356b-c2t9b\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:22.497097 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.497066 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:22.874443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:22.874417 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b"] Nov 02 23:49:22.877293 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:49:22.877262 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bafe2a_66b3_4e3b_a1dd_dacc2f72ed2f.slice/crio-4f8535eaf39c2c366fd2241230e1e8fb2b51e4bb59ac1971df8f15d0e1e1c62c WatchSource:0}: Error finding container 4f8535eaf39c2c366fd2241230e1e8fb2b51e4bb59ac1971df8f15d0e1e1c62c: Status 404 returned error can't find the container with id 4f8535eaf39c2c366fd2241230e1e8fb2b51e4bb59ac1971df8f15d0e1e1c62c Nov 02 23:49:23.288465 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:23.288424 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" event={"ID":"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f","Type":"ContainerStarted","Data":"c23b7a9c13e57ba71e87c29dd05e2edf787dfb825ced6d7ffee73f346ec60b86"} Nov 02 23:49:23.288888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:23.288485 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" event={"ID":"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f","Type":"ContainerStarted","Data":"4f8535eaf39c2c366fd2241230e1e8fb2b51e4bb59ac1971df8f15d0e1e1c62c"} Nov 02 23:49:23.445177 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:23.445039 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" podStartSLOduration=1.445020307 podStartE2EDuration="1.445020307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:23.395460456 +0000 UTC m=+3356.802772303" watchObservedRunningTime="2026-11-02 23:49:23.445020307 +0000 UTC m=+3356.852332138" Nov 02 23:49:24.296062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:24.296025 5672 generic.go:356] "Generic (PLEG): container finished" podID="e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f" containerID="c23b7a9c13e57ba71e87c29dd05e2edf787dfb825ced6d7ffee73f346ec60b86" exitCode=0 Nov 02 23:49:24.296549 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:24.296110 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" event={"ID":"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f","Type":"ContainerDied","Data":"c23b7a9c13e57ba71e87c29dd05e2edf787dfb825ced6d7ffee73f346ec60b86"} Nov 02 23:49:25.465808 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:25.465776 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:25.531306 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:25.531272 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-xks-ca\") pod \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " Nov 02 23:49:25.531485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:25.531342 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-kube-api-access-mm76f\" (UniqueName: \"kubernetes.io/projected/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-kube-api-access-mm76f\") pod \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " Nov 02 23:49:25.531485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:25.531443 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-openshift-ca\") pod \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\" (UID: \"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f\") " Nov 02 23:49:25.531873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:25.531837 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-openshift-ca" pod "e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f" (UID: "e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:49:25.533542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:25.533515 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-kube-api-access-mm76f" pod "e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f" (UID: "e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f"). InnerVolumeSpecName "kube-api-access-mm76f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:49:25.533686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:25.533582 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-xks-ca" pod "e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f" (UID: "e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:49:25.633793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:25.633711 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:25.633793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:25.633750 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm76f\" (UniqueName: \"kubernetes.io/projected/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-kube-api-access-mm76f\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:25.633793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:25.633766 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:26.020010 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:49:26.019976 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:49:26.310638 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:26.310561 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" Nov 02 23:49:26.310638 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:26.310588 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b" event={"ID":"e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f","Type":"ContainerDied","Data":"4f8535eaf39c2c366fd2241230e1e8fb2b51e4bb59ac1971df8f15d0e1e1c62c"} Nov 02 23:49:26.310833 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:26.310641 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f8535eaf39c2c366fd2241230e1e8fb2b51e4bb59ac1971df8f15d0e1e1c62c" Nov 02 23:49:32.953460 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:32.953425 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465"] Nov 02 23:49:32.953959 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:32.953944 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f" containerName="revoke-keys" Nov 02 23:49:32.954007 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:32.953960 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f" containerName="revoke-keys" Nov 02 23:49:32.954065 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:32.954055 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f" containerName="revoke-keys" Nov 02 23:49:32.958641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:32.958619 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:33.010396 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.010365 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465"] Nov 02 23:49:33.057539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.056418 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:49:33.057539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.056600 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:49:33.057539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.057372 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:49:33.057539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.057459 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:49:33.114056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.114026 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1598c534-hc465\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:33.114262 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.114064 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1598c534-hc465\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:33.114262 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.114186 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krldj\" (UniqueName: \"kubernetes.io/projected/4a894456-5e05-4f39-86e2-54ce5e24f679-kube-api-access-krldj\") pod \"maas-api-revoke-sub-1598c534-hc465\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:33.215797 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.215711 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1598c534-hc465\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:33.215797 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.215765 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1598c534-hc465\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:33.216030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.215828 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krldj\" (UniqueName: \"kubernetes.io/projected/4a894456-5e05-4f39-86e2-54ce5e24f679-kube-api-access-krldj\") pod \"maas-api-revoke-sub-1598c534-hc465\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:33.216434 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.216406 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1598c534-hc465\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:33.218055 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.218025 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1598c534-hc465\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:33.269962 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.269927 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krldj\" (UniqueName: \"kubernetes.io/projected/4a894456-5e05-4f39-86e2-54ce5e24f679-kube-api-access-krldj\") pod \"maas-api-revoke-sub-1598c534-hc465\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:33.275391 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.275361 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:33.644947 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:33.644908 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465"] Nov 02 23:49:33.645289 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:49:33.645258 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a894456_5e05_4f39_86e2_54ce5e24f679.slice/crio-7bebfd19b91926a2da6ada0262784c6155161196ad7c06e5336550b36530a7e8 WatchSource:0}: Error finding container 7bebfd19b91926a2da6ada0262784c6155161196ad7c06e5336550b36530a7e8: Status 404 returned error can't find the container with id 7bebfd19b91926a2da6ada0262784c6155161196ad7c06e5336550b36530a7e8 Nov 02 23:49:34.366343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:34.366298 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" event={"ID":"4a894456-5e05-4f39-86e2-54ce5e24f679","Type":"ContainerStarted","Data":"009c4231d978c87c9146f2472b2bedd63c368ca5fca9620954fb366dcba896e1"} Nov 02 23:49:34.366773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:34.366364 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" event={"ID":"4a894456-5e05-4f39-86e2-54ce5e24f679","Type":"ContainerStarted","Data":"7bebfd19b91926a2da6ada0262784c6155161196ad7c06e5336550b36530a7e8"} Nov 02 23:49:34.476338 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:34.476272 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" podStartSLOduration=2.476252841 podStartE2EDuration="2.476252841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:34.471953588 +0000 UTC m=+3367.879265421" watchObservedRunningTime="2026-11-02 23:49:34.476252841 +0000 UTC m=+3367.883564675" Nov 02 23:49:35.373293 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:35.373259 5672 generic.go:356] "Generic (PLEG): container finished" podID="4a894456-5e05-4f39-86e2-54ce5e24f679" containerID="009c4231d978c87c9146f2472b2bedd63c368ca5fca9620954fb366dcba896e1" exitCode=0 Nov 02 23:49:35.373677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:35.373340 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" event={"ID":"4a894456-5e05-4f39-86e2-54ce5e24f679","Type":"ContainerDied","Data":"009c4231d978c87c9146f2472b2bedd63c368ca5fca9620954fb366dcba896e1"} Nov 02 23:49:36.541033 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:36.541002 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:36.654891 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:36.653683 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-xks-ca\") pod \"4a894456-5e05-4f39-86e2-54ce5e24f679\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " Nov 02 23:49:36.654891 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:36.653795 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a894456-5e05-4f39-86e2-54ce5e24f679-kube-api-access-krldj\" (UniqueName: \"kubernetes.io/projected/4a894456-5e05-4f39-86e2-54ce5e24f679-kube-api-access-krldj\") pod \"4a894456-5e05-4f39-86e2-54ce5e24f679\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " Nov 02 23:49:36.654891 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:36.653912 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-openshift-ca\") pod \"4a894456-5e05-4f39-86e2-54ce5e24f679\" (UID: \"4a894456-5e05-4f39-86e2-54ce5e24f679\") " Nov 02 23:49:36.657954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:36.657624 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-openshift-ca" pod "4a894456-5e05-4f39-86e2-54ce5e24f679" (UID: "4a894456-5e05-4f39-86e2-54ce5e24f679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:49:36.661050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:36.660693 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a894456-5e05-4f39-86e2-54ce5e24f679-kube-api-access-krldj" pod "4a894456-5e05-4f39-86e2-54ce5e24f679" (UID: "4a894456-5e05-4f39-86e2-54ce5e24f679"). InnerVolumeSpecName "kube-api-access-krldj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:49:36.673898 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:36.673847 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-xks-ca" pod "4a894456-5e05-4f39-86e2-54ce5e24f679" (UID: "4a894456-5e05-4f39-86e2-54ce5e24f679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:49:36.755387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:36.755358 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krldj\" (UniqueName: \"kubernetes.io/projected/4a894456-5e05-4f39-86e2-54ce5e24f679-kube-api-access-krldj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:36.755387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:36.755383 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:36.755387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:36.755393 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a894456-5e05-4f39-86e2-54ce5e24f679-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:49:37.387522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:37.387493 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" Nov 02 23:49:37.387719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:37.387539 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465" event={"ID":"4a894456-5e05-4f39-86e2-54ce5e24f679","Type":"ContainerDied","Data":"7bebfd19b91926a2da6ada0262784c6155161196ad7c06e5336550b36530a7e8"} Nov 02 23:49:37.387719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:37.387599 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bebfd19b91926a2da6ada0262784c6155161196ad7c06e5336550b36530a7e8" Nov 02 23:49:46.204086 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.204050 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r"] Nov 02 23:49:46.204567 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.204550 5672 cpu_manager.go:414] "containerMap: removing container" podUID="4a894456-5e05-4f39-86e2-54ce5e24f679" containerName="revoke-keys" Nov 02 23:49:46.204624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.204568 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a894456-5e05-4f39-86e2-54ce5e24f679" containerName="revoke-keys" Nov 02 23:49:46.204685 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.204676 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a894456-5e05-4f39-86e2-54ce5e24f679" containerName="revoke-keys" Nov 02 23:49:46.207941 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.207923 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:49:46.261397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.261361 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r"] Nov 02 23:49:46.266710 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.266681 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfbgv\" (UniqueName: \"kubernetes.io/projected/41ee9e23-5087-46d4-92d6-52b1e454a88b-kube-api-access-dfbgv\") pod \"payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r\" (UID: \"41ee9e23-5087-46d4-92d6-52b1e454a88b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:49:46.266888 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.266825 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41ee9e23-5087-46d4-92d6-52b1e454a88b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r\" (UID: \"41ee9e23-5087-46d4-92d6-52b1e454a88b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:49:46.303441 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.303384 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8c2bff\"" Nov 02 23:49:46.303651 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.303442 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8c2bff-dockercfg-x6gf2\"" Nov 02 23:49:46.367542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.367509 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbgv\" (UniqueName: \"kubernetes.io/projected/41ee9e23-5087-46d4-92d6-52b1e454a88b-kube-api-access-dfbgv\") pod \"payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r\" (UID: \"41ee9e23-5087-46d4-92d6-52b1e454a88b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:49:46.367731 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.367612 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41ee9e23-5087-46d4-92d6-52b1e454a88b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r\" (UID: \"41ee9e23-5087-46d4-92d6-52b1e454a88b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:49:46.368402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.368375 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41ee9e23-5087-46d4-92d6-52b1e454a88b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r\" (UID: \"41ee9e23-5087-46d4-92d6-52b1e454a88b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:49:46.422843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.422810 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbgv\" (UniqueName: \"kubernetes.io/projected/41ee9e23-5087-46d4-92d6-52b1e454a88b-kube-api-access-dfbgv\") pod \"payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r\" (UID: \"41ee9e23-5087-46d4-92d6-52b1e454a88b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:49:46.521126 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.521098 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:49:46.897690 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:46.897659 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r"] Nov 02 23:49:46.898942 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:49:46.898910 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ee9e23_5087_46d4_92d6_52b1e454a88b.slice/crio-792d562da04965c88ef8edbf6d87695d1460b73f4553d743b11b2034d177cc8f WatchSource:0}: Error finding container 792d562da04965c88ef8edbf6d87695d1460b73f4553d743b11b2034d177cc8f: Status 404 returned error can't find the container with id 792d562da04965c88ef8edbf6d87695d1460b73f4553d743b11b2034d177cc8f Nov 02 23:49:47.475377 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:47.475332 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" event={"ID":"41ee9e23-5087-46d4-92d6-52b1e454a88b","Type":"ContainerStarted","Data":"0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a"} Nov 02 23:49:47.475879 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:47.475397 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" event={"ID":"41ee9e23-5087-46d4-92d6-52b1e454a88b","Type":"ContainerStarted","Data":"792d562da04965c88ef8edbf6d87695d1460b73f4553d743b11b2034d177cc8f"} Nov 02 23:49:47.475879 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:47.475440 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:49:47.585329 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:47.585278 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" podStartSLOduration=1.5852657799999998 podStartE2EDuration="1.58526578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:47.583013926 +0000 UTC m=+3380.990325756" watchObservedRunningTime="2026-11-02 23:49:47.58526578 +0000 UTC m=+3380.992577610" Nov 02 23:49:51.826327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:51.826289 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:49:56.020536 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:49:56.020498 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:49:57.931601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:57.931561 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp"] Nov 02 23:49:57.935390 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:57.935365 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:49:58.002509 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.002461 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83c0e838-kkgwp\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:49:58.002684 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.002599 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83c0e838-kkgwp\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:49:58.002843 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.002816 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbhsv\" (UniqueName: \"kubernetes.io/projected/3da40d92-aae0-46c7-b744-84a0a7575158-kube-api-access-bbhsv\") pod \"maas-api-revoke-sub-83c0e838-kkgwp\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:49:58.030968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.030933 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:49:58.031185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.031065 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:49:58.031185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.031142 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:49:58.031325 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.031288 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:49:58.035443 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.035416 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp"] Nov 02 23:49:58.104588 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.104556 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbhsv\" (UniqueName: \"kubernetes.io/projected/3da40d92-aae0-46c7-b744-84a0a7575158-kube-api-access-bbhsv\") pod \"maas-api-revoke-sub-83c0e838-kkgwp\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:49:58.104782 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.104629 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83c0e838-kkgwp\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:49:58.104782 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.104684 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83c0e838-kkgwp\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:49:58.105384 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.105355 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83c0e838-kkgwp\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:49:58.106993 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.106973 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83c0e838-kkgwp\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:49:58.158563 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.158531 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbhsv\" (UniqueName: \"kubernetes.io/projected/3da40d92-aae0-46c7-b744-84a0a7575158-kube-api-access-bbhsv\") pod \"maas-api-revoke-sub-83c0e838-kkgwp\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:49:58.251214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.251174 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:49:58.485188 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.485143 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:49:58.653730 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:58.652880 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp"] Nov 02 23:49:59.559705 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:59.559672 5672 generic.go:356] "Generic (PLEG): container finished" podID="3da40d92-aae0-46c7-b744-84a0a7575158" containerID="e8d3c1ac22439762468a4e60b349a40b653e145a39fbfc3b7f178b642779f97f" exitCode=0 Nov 02 23:49:59.560067 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:59.559731 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" event={"ID":"3da40d92-aae0-46c7-b744-84a0a7575158","Type":"ContainerDied","Data":"e8d3c1ac22439762468a4e60b349a40b653e145a39fbfc3b7f178b642779f97f"} Nov 02 23:49:59.560067 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:49:59.559782 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" event={"ID":"3da40d92-aae0-46c7-b744-84a0a7575158","Type":"ContainerStarted","Data":"71eb423b62ef2bbf9aea06b68cce5627b13afe21663448ed3ee5e2cdc2ab856e"} Nov 02 23:50:00.709935 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:00.709905 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:50:00.834243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:00.834125 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-xks-ca\") pod \"3da40d92-aae0-46c7-b744-84a0a7575158\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " Nov 02 23:50:00.834432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:00.834291 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da40d92-aae0-46c7-b744-84a0a7575158-kube-api-access-bbhsv\" (UniqueName: \"kubernetes.io/projected/3da40d92-aae0-46c7-b744-84a0a7575158-kube-api-access-bbhsv\") pod \"3da40d92-aae0-46c7-b744-84a0a7575158\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " Nov 02 23:50:00.834432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:00.834358 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-openshift-ca\") pod \"3da40d92-aae0-46c7-b744-84a0a7575158\" (UID: \"3da40d92-aae0-46c7-b744-84a0a7575158\") " Nov 02 23:50:00.835098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:00.835053 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-openshift-ca" pod "3da40d92-aae0-46c7-b744-84a0a7575158" (UID: "3da40d92-aae0-46c7-b744-84a0a7575158"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:50:00.836860 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:00.836833 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-xks-ca" pod "3da40d92-aae0-46c7-b744-84a0a7575158" (UID: "3da40d92-aae0-46c7-b744-84a0a7575158"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:50:00.837139 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:00.837106 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da40d92-aae0-46c7-b744-84a0a7575158-kube-api-access-bbhsv" pod "3da40d92-aae0-46c7-b744-84a0a7575158" (UID: "3da40d92-aae0-46c7-b744-84a0a7575158"). InnerVolumeSpecName "kube-api-access-bbhsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:50:00.935756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:00.935722 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:00.935893 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:00.935760 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbhsv\" (UniqueName: \"kubernetes.io/projected/3da40d92-aae0-46c7-b744-84a0a7575158-kube-api-access-bbhsv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:00.935893 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:00.935775 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da40d92-aae0-46c7-b744-84a0a7575158-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:01.576185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:01.576131 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" Nov 02 23:50:01.576372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:01.576192 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp" event={"ID":"3da40d92-aae0-46c7-b744-84a0a7575158","Type":"ContainerDied","Data":"71eb423b62ef2bbf9aea06b68cce5627b13afe21663448ed3ee5e2cdc2ab856e"} Nov 02 23:50:01.576372 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:01.576276 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71eb423b62ef2bbf9aea06b68cce5627b13afe21663448ed3ee5e2cdc2ab856e" Nov 02 23:50:08.636760 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.636727 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5"] Nov 02 23:50:08.637261 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.637247 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3da40d92-aae0-46c7-b744-84a0a7575158" containerName="revoke-keys" Nov 02 23:50:08.637314 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.637262 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da40d92-aae0-46c7-b744-84a0a7575158" containerName="revoke-keys" Nov 02 23:50:08.637376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.637367 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da40d92-aae0-46c7-b744-84a0a7575158" containerName="revoke-keys" Nov 02 23:50:08.640969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.640948 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:08.689998 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.689970 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:50:08.693202 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.693175 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5"] Nov 02 23:50:08.727351 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.727321 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4696bedd-zsvn5\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:08.727525 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.727369 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrvt\" (UniqueName: \"kubernetes.io/projected/a302741c-5165-4dbe-ad80-cd06c96c54e8-kube-api-access-flrvt\") pod \"maas-api-revoke-sub-4696bedd-zsvn5\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:08.727525 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.727511 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4696bedd-zsvn5\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:08.736098 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.736073 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:50:08.736220 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.736207 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:50:08.736279 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.736227 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:50:08.828525 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.828493 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4696bedd-zsvn5\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:08.828698 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.828576 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4696bedd-zsvn5\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:08.828698 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.828608 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrvt\" (UniqueName: \"kubernetes.io/projected/a302741c-5165-4dbe-ad80-cd06c96c54e8-kube-api-access-flrvt\") pod \"maas-api-revoke-sub-4696bedd-zsvn5\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:08.829221 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.829200 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4696bedd-zsvn5\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:08.830849 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.830826 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4696bedd-zsvn5\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:08.881402 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.881375 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrvt\" (UniqueName: \"kubernetes.io/projected/a302741c-5165-4dbe-ad80-cd06c96c54e8-kube-api-access-flrvt\") pod \"maas-api-revoke-sub-4696bedd-zsvn5\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:08.954554 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:08.954473 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:09.323287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:09.323257 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5"] Nov 02 23:50:09.324733 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:50:09.324703 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda302741c_5165_4dbe_ad80_cd06c96c54e8.slice/crio-a529818352c550c92584800962a7f5b555c16e9b7d15d1afef554a3b5b5c1ff7 WatchSource:0}: Error finding container a529818352c550c92584800962a7f5b555c16e9b7d15d1afef554a3b5b5c1ff7: Status 404 returned error can't find the container with id a529818352c550c92584800962a7f5b555c16e9b7d15d1afef554a3b5b5c1ff7 Nov 02 23:50:09.629800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:09.629765 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" event={"ID":"a302741c-5165-4dbe-ad80-cd06c96c54e8","Type":"ContainerStarted","Data":"a529818352c550c92584800962a7f5b555c16e9b7d15d1afef554a3b5b5c1ff7"} Nov 02 23:50:10.638469 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:10.638381 5672 generic.go:356] "Generic (PLEG): container finished" podID="a302741c-5165-4dbe-ad80-cd06c96c54e8" containerID="dbc95b6241da3dbc0b93e3400514e5415df10e4b3ff2206e047fe781df197e99" exitCode=0 Nov 02 23:50:10.638839 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:10.638460 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" event={"ID":"a302741c-5165-4dbe-ad80-cd06c96c54e8","Type":"ContainerDied","Data":"dbc95b6241da3dbc0b93e3400514e5415df10e4b3ff2206e047fe781df197e99"} Nov 02 23:50:11.789207 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:11.789182 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:11.858137 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:11.858107 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-xks-ca\") pod \"a302741c-5165-4dbe-ad80-cd06c96c54e8\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " Nov 02 23:50:11.858291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:11.858158 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-openshift-ca\") pod \"a302741c-5165-4dbe-ad80-cd06c96c54e8\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " Nov 02 23:50:11.858291 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:11.858276 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a302741c-5165-4dbe-ad80-cd06c96c54e8-kube-api-access-flrvt\" (UniqueName: \"kubernetes.io/projected/a302741c-5165-4dbe-ad80-cd06c96c54e8-kube-api-access-flrvt\") pod \"a302741c-5165-4dbe-ad80-cd06c96c54e8\" (UID: \"a302741c-5165-4dbe-ad80-cd06c96c54e8\") " Nov 02 23:50:11.858563 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:11.858537 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-openshift-ca" pod "a302741c-5165-4dbe-ad80-cd06c96c54e8" (UID: "a302741c-5165-4dbe-ad80-cd06c96c54e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:50:11.860266 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:11.860241 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-xks-ca" pod "a302741c-5165-4dbe-ad80-cd06c96c54e8" (UID: "a302741c-5165-4dbe-ad80-cd06c96c54e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:50:11.860360 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:11.860315 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a302741c-5165-4dbe-ad80-cd06c96c54e8-kube-api-access-flrvt" pod "a302741c-5165-4dbe-ad80-cd06c96c54e8" (UID: "a302741c-5165-4dbe-ad80-cd06c96c54e8"). InnerVolumeSpecName "kube-api-access-flrvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:50:11.959187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:11.959092 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:11.959187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:11.959121 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a302741c-5165-4dbe-ad80-cd06c96c54e8-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:11.959187 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:11.959131 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flrvt\" (UniqueName: \"kubernetes.io/projected/a302741c-5165-4dbe-ad80-cd06c96c54e8-kube-api-access-flrvt\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:12.652108 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:12.652083 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" Nov 02 23:50:12.652314 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:12.652121 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5" event={"ID":"a302741c-5165-4dbe-ad80-cd06c96c54e8","Type":"ContainerDied","Data":"a529818352c550c92584800962a7f5b555c16e9b7d15d1afef554a3b5b5c1ff7"} Nov 02 23:50:12.652314 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:12.652191 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a529818352c550c92584800962a7f5b555c16e9b7d15d1afef554a3b5b5c1ff7" Nov 02 23:50:20.135661 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:20.135622 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8"] Nov 02 23:50:26.020988 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:50:26.020945 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:50:32.223049 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.223010 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr"] Nov 02 23:50:32.223611 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.223596 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a302741c-5165-4dbe-ad80-cd06c96c54e8" containerName="revoke-keys" Nov 02 23:50:32.223658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.223613 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a302741c-5165-4dbe-ad80-cd06c96c54e8" containerName="revoke-keys" Nov 02 23:50:32.223725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.223715 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a302741c-5165-4dbe-ad80-cd06c96c54e8" containerName="revoke-keys" Nov 02 23:50:32.227745 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.227722 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:32.280812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.280777 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr"] Nov 02 23:50:32.304489 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.304451 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d03e78-p6xmr\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:32.304669 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.304510 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7x5\" (UniqueName: \"kubernetes.io/projected/7193049d-1622-4e94-93f4-ac729c1d1d97-kube-api-access-tx7x5\") pod \"maas-api-revoke-sub-75d03e78-p6xmr\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:32.304756 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.304677 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d03e78-p6xmr\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:32.319625 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.319596 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:50:32.319780 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.319720 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:50:32.322650 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.322631 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:50:32.322943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.322927 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:50:32.405418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.405391 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d03e78-p6xmr\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:32.405599 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.405444 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d03e78-p6xmr\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:32.405599 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.405502 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7x5\" (UniqueName: \"kubernetes.io/projected/7193049d-1622-4e94-93f4-ac729c1d1d97-kube-api-access-tx7x5\") pod \"maas-api-revoke-sub-75d03e78-p6xmr\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:32.406246 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.406226 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d03e78-p6xmr\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:32.407710 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.407691 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d03e78-p6xmr\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:32.458173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.458122 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7x5\" (UniqueName: \"kubernetes.io/projected/7193049d-1622-4e94-93f4-ac729c1d1d97-kube-api-access-tx7x5\") pod \"maas-api-revoke-sub-75d03e78-p6xmr\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:32.543013 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.542931 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:32.909594 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:32.909557 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr"] Nov 02 23:50:32.911194 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:50:32.911136 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7193049d_1622_4e94_93f4_ac729c1d1d97.slice/crio-51152cc08bc92132684d34632c567abe41bd4561167e747a95c90691e3527a72 WatchSource:0}: Error finding container 51152cc08bc92132684d34632c567abe41bd4561167e747a95c90691e3527a72: Status 404 returned error can't find the container with id 51152cc08bc92132684d34632c567abe41bd4561167e747a95c90691e3527a72 Nov 02 23:50:33.796122 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:33.796092 5672 generic.go:356] "Generic (PLEG): container finished" podID="7193049d-1622-4e94-93f4-ac729c1d1d97" containerID="1ddf9df3ee7a7328bfb0447d01985528148d98d9466096bee430051227a76cd9" exitCode=0 Nov 02 23:50:33.796510 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:33.796190 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" event={"ID":"7193049d-1622-4e94-93f4-ac729c1d1d97","Type":"ContainerDied","Data":"1ddf9df3ee7a7328bfb0447d01985528148d98d9466096bee430051227a76cd9"} Nov 02 23:50:33.796510 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:33.796256 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" event={"ID":"7193049d-1622-4e94-93f4-ac729c1d1d97","Type":"ContainerStarted","Data":"51152cc08bc92132684d34632c567abe41bd4561167e747a95c90691e3527a72"} Nov 02 23:50:34.957804 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:34.957775 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:35.037281 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.037251 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-xks-ca\") pod \"7193049d-1622-4e94-93f4-ac729c1d1d97\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " Nov 02 23:50:35.037452 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.037351 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7193049d-1622-4e94-93f4-ac729c1d1d97-kube-api-access-tx7x5\" (UniqueName: \"kubernetes.io/projected/7193049d-1622-4e94-93f4-ac729c1d1d97-kube-api-access-tx7x5\") pod \"7193049d-1622-4e94-93f4-ac729c1d1d97\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " Nov 02 23:50:35.037452 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.037412 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-openshift-ca\") pod \"7193049d-1622-4e94-93f4-ac729c1d1d97\" (UID: \"7193049d-1622-4e94-93f4-ac729c1d1d97\") " Nov 02 23:50:35.037774 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.037745 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-openshift-ca" pod "7193049d-1622-4e94-93f4-ac729c1d1d97" (UID: "7193049d-1622-4e94-93f4-ac729c1d1d97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:50:35.037938 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.037914 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:35.039460 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.039437 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-xks-ca" pod "7193049d-1622-4e94-93f4-ac729c1d1d97" (UID: "7193049d-1622-4e94-93f4-ac729c1d1d97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:50:35.039563 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.039517 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7193049d-1622-4e94-93f4-ac729c1d1d97-kube-api-access-tx7x5" pod "7193049d-1622-4e94-93f4-ac729c1d1d97" (UID: "7193049d-1622-4e94-93f4-ac729c1d1d97"). InnerVolumeSpecName "kube-api-access-tx7x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:50:35.139477 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.139398 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx7x5\" (UniqueName: \"kubernetes.io/projected/7193049d-1622-4e94-93f4-ac729c1d1d97-kube-api-access-tx7x5\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:35.139477 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.139426 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7193049d-1622-4e94-93f4-ac729c1d1d97-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:35.729491 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.729397 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" podUID="105a58d0-a50d-4ef8-a450-4778c6b97201" containerName="main" containerID="cri-o://2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6" gracePeriod=45 Nov 02 23:50:35.810490 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.810454 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" event={"ID":"7193049d-1622-4e94-93f4-ac729c1d1d97","Type":"ContainerDied","Data":"51152cc08bc92132684d34632c567abe41bd4561167e747a95c90691e3527a72"} Nov 02 23:50:35.810490 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.810481 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr" Nov 02 23:50:35.810700 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:35.810507 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51152cc08bc92132684d34632c567abe41bd4561167e747a95c90691e3527a72" Nov 02 23:50:36.005056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.005024 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:50:36.150128 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.150096 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/105a58d0-a50d-4ef8-a450-4778c6b97201-tls-certs\" (UniqueName: \"kubernetes.io/secret/105a58d0-a50d-4ef8-a450-4778c6b97201-tls-certs\") pod \"105a58d0-a50d-4ef8-a450-4778c6b97201\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " Nov 02 23:50:36.150357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.150199 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-model-cache\") pod \"105a58d0-a50d-4ef8-a450-4778c6b97201\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " Nov 02 23:50:36.150357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.150240 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105a58d0-a50d-4ef8-a450-4778c6b97201-kube-api-access-mbh59\" (UniqueName: \"kubernetes.io/projected/105a58d0-a50d-4ef8-a450-4778c6b97201-kube-api-access-mbh59\") pod \"105a58d0-a50d-4ef8-a450-4778c6b97201\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " Nov 02 23:50:36.150357 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.150324 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-home\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-home\") pod \"105a58d0-a50d-4ef8-a450-4778c6b97201\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " Nov 02 23:50:36.150545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.150399 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-tmp-dir\") pod \"105a58d0-a50d-4ef8-a450-4778c6b97201\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " Nov 02 23:50:36.150545 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.150482 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-dshm\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-dshm\") pod \"105a58d0-a50d-4ef8-a450-4778c6b97201\" (UID: \"105a58d0-a50d-4ef8-a450-4778c6b97201\") " Nov 02 23:50:36.151144 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.151113 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-tmp-dir" pod "105a58d0-a50d-4ef8-a450-4778c6b97201" (UID: "105a58d0-a50d-4ef8-a450-4778c6b97201"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:50:36.151414 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.151351 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-home" pod "105a58d0-a50d-4ef8-a450-4778c6b97201" (UID: "105a58d0-a50d-4ef8-a450-4778c6b97201"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:50:36.151621 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.151577 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-model-cache" pod "105a58d0-a50d-4ef8-a450-4778c6b97201" (UID: "105a58d0-a50d-4ef8-a450-4778c6b97201"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:50:36.156754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.156699 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105a58d0-a50d-4ef8-a450-4778c6b97201-tls-certs" pod "105a58d0-a50d-4ef8-a450-4778c6b97201" (UID: "105a58d0-a50d-4ef8-a450-4778c6b97201"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:50:36.156754 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.156709 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-dshm" pod "105a58d0-a50d-4ef8-a450-4778c6b97201" (UID: "105a58d0-a50d-4ef8-a450-4778c6b97201"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:50:36.156969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.156826 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105a58d0-a50d-4ef8-a450-4778c6b97201-kube-api-access-mbh59" pod "105a58d0-a50d-4ef8-a450-4778c6b97201" (UID: "105a58d0-a50d-4ef8-a450-4778c6b97201"). InnerVolumeSpecName "kube-api-access-mbh59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:50:36.252079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.251981 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:36.252079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.252018 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:36.252079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.252033 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:36.252079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.252046 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105a58d0-a50d-4ef8-a450-4778c6b97201-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:36.252079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.252058 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/105a58d0-a50d-4ef8-a450-4778c6b97201-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:36.252079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.252074 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbh59\" (UniqueName: \"kubernetes.io/projected/105a58d0-a50d-4ef8-a450-4778c6b97201-kube-api-access-mbh59\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:36.818347 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.818310 5672 generic.go:356] "Generic (PLEG): container finished" podID="105a58d0-a50d-4ef8-a450-4778c6b97201" containerID="2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6" exitCode=0 Nov 02 23:50:36.818539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.818372 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" Nov 02 23:50:36.818539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.818399 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" event={"ID":"105a58d0-a50d-4ef8-a450-4778c6b97201","Type":"ContainerDied","Data":"2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6"} Nov 02 23:50:36.818539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.818441 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8" event={"ID":"105a58d0-a50d-4ef8-a450-4778c6b97201","Type":"ContainerDied","Data":"fb14ca17f612f35f722b041b4b9e0232536020117fa50943c024efbaf1241b30"} Nov 02 23:50:36.818539 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.818458 5672 scope.go:122] "RemoveContainer" containerID="2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6" Nov 02 23:50:36.827977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.827959 5672 scope.go:122] "RemoveContainer" containerID="2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6" Nov 02 23:50:36.828374 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:50:36.828353 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6\": container with ID starting with 2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6 not found: ID does not exist" containerID="2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6" Nov 02 23:50:36.828439 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.828384 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6"} err="failed to get container status \"2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6\": rpc error: code = NotFound desc = could not find container \"2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6\": container with ID starting with 2bd623e62bfa8bc2f6fe00d9ba6664c1f1471b39938c9b56713e2b41d866e1b6 not found: ID does not exist" Nov 02 23:50:36.971415 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.971383 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8"] Nov 02 23:50:36.978714 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:36.977167 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/auth-test-model-c25940-kserve-6fdf77cd89-5cmx8"] Nov 02 23:50:38.715047 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:38.715000 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105a58d0-a50d-4ef8-a450-4778c6b97201" path="/var/lib/kubelet/pods/105a58d0-a50d-4ef8-a450-4778c6b97201/volumes" Nov 02 23:50:42.946578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:42.946539 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz"] Nov 02 23:50:42.947079 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:42.947062 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7193049d-1622-4e94-93f4-ac729c1d1d97" containerName="revoke-keys" Nov 02 23:50:42.947138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:42.947080 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7193049d-1622-4e94-93f4-ac729c1d1d97" containerName="revoke-keys" Nov 02 23:50:42.947138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:42.947102 5672 cpu_manager.go:414] "containerMap: removing container" podUID="105a58d0-a50d-4ef8-a450-4778c6b97201" containerName="main" Nov 02 23:50:42.947138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:42.947107 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105a58d0-a50d-4ef8-a450-4778c6b97201" containerName="main" Nov 02 23:50:42.947293 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:42.947245 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7193049d-1622-4e94-93f4-ac729c1d1d97" containerName="revoke-keys" Nov 02 23:50:42.947293 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:42.947257 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="105a58d0-a50d-4ef8-a450-4778c6b97201" containerName="main" Nov 02 23:50:42.951982 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:42.951962 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:43.003740 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.003705 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz"] Nov 02 23:50:43.021811 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.021778 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce38609-7sgtz\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:43.021966 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.021832 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce38609-7sgtz\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:43.022056 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.022001 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5zb\" (UniqueName: \"kubernetes.io/projected/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-kube-api-access-mm5zb\") pod \"maas-api-revoke-sub-6ce38609-7sgtz\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:43.032324 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.032295 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:50:43.032440 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.032341 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:50:43.032440 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.032375 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:50:43.046828 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.046807 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:50:43.122836 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.122803 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce38609-7sgtz\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:43.122991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.122865 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce38609-7sgtz\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:43.122991 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.122926 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5zb\" (UniqueName: \"kubernetes.io/projected/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-kube-api-access-mm5zb\") pod \"maas-api-revoke-sub-6ce38609-7sgtz\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:43.123548 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.123523 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce38609-7sgtz\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:43.125311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.125287 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce38609-7sgtz\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:43.176690 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.176658 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5zb\" (UniqueName: \"kubernetes.io/projected/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-kube-api-access-mm5zb\") pod \"maas-api-revoke-sub-6ce38609-7sgtz\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:43.268977 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.268946 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:43.638503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.638268 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz"] Nov 02 23:50:43.641588 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:50:43.641557 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aa4b8b_e6df_4e19_8fd9_c352394b5fcb.slice/crio-a9fdcd46c330511a0a348182e725d37694236870096e0fddc6a8fcbc93a2d198 WatchSource:0}: Error finding container a9fdcd46c330511a0a348182e725d37694236870096e0fddc6a8fcbc93a2d198: Status 404 returned error can't find the container with id a9fdcd46c330511a0a348182e725d37694236870096e0fddc6a8fcbc93a2d198 Nov 02 23:50:43.878172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:43.878121 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" event={"ID":"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb","Type":"ContainerStarted","Data":"a9fdcd46c330511a0a348182e725d37694236870096e0fddc6a8fcbc93a2d198"} Nov 02 23:50:44.886485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:44.886397 5672 generic.go:356] "Generic (PLEG): container finished" podID="84aa4b8b-e6df-4e19-8fd9-c352394b5fcb" containerID="149f80064724c4c71b62057612ee556b9761c5477ecbc0b6458a98c2f0e4075c" exitCode=0 Nov 02 23:50:44.886860 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:44.886484 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" event={"ID":"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb","Type":"ContainerDied","Data":"149f80064724c4c71b62057612ee556b9761c5477ecbc0b6458a98c2f0e4075c"} Nov 02 23:50:46.075853 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.075824 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:46.156336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.156258 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-kube-api-access-mm5zb\" (UniqueName: \"kubernetes.io/projected/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-kube-api-access-mm5zb\") pod \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " Nov 02 23:50:46.156503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.156455 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-openshift-ca\") pod \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " Nov 02 23:50:46.156503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.156493 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-xks-ca\") pod \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\" (UID: \"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb\") " Nov 02 23:50:46.156875 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.156848 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-openshift-ca" pod "84aa4b8b-e6df-4e19-8fd9-c352394b5fcb" (UID: "84aa4b8b-e6df-4e19-8fd9-c352394b5fcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:50:46.158487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.158457 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-kube-api-access-mm5zb" pod "84aa4b8b-e6df-4e19-8fd9-c352394b5fcb" (UID: "84aa4b8b-e6df-4e19-8fd9-c352394b5fcb"). InnerVolumeSpecName "kube-api-access-mm5zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:50:46.158590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.158515 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-xks-ca" pod "84aa4b8b-e6df-4e19-8fd9-c352394b5fcb" (UID: "84aa4b8b-e6df-4e19-8fd9-c352394b5fcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:50:46.257914 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.257876 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:46.257914 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.257916 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:46.258114 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.257933 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm5zb\" (UniqueName: \"kubernetes.io/projected/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb-kube-api-access-mm5zb\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:50:46.899867 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.899832 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" event={"ID":"84aa4b8b-e6df-4e19-8fd9-c352394b5fcb","Type":"ContainerDied","Data":"a9fdcd46c330511a0a348182e725d37694236870096e0fddc6a8fcbc93a2d198"} Nov 02 23:50:46.900031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.899887 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9fdcd46c330511a0a348182e725d37694236870096e0fddc6a8fcbc93a2d198" Nov 02 23:50:46.900031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:46.899845 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz" Nov 02 23:50:48.250350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:50:48.250312 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:50:56.020631 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:50:56.020594 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:51:01.100523 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.100485 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd"] Nov 02 23:51:01.101101 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.101074 5672 cpu_manager.go:414] "containerMap: removing container" podUID="84aa4b8b-e6df-4e19-8fd9-c352394b5fcb" containerName="revoke-keys" Nov 02 23:51:01.101101 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.101099 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aa4b8b-e6df-4e19-8fd9-c352394b5fcb" containerName="revoke-keys" Nov 02 23:51:01.101295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.101258 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aa4b8b-e6df-4e19-8fd9-c352394b5fcb" containerName="revoke-keys" Nov 02 23:51:01.104356 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.104338 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.154283 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.153630 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f2df33-openshift-default-dockercfg-fhg27\"" Nov 02 23:51:01.204504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.204470 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd"] Nov 02 23:51:01.232913 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.232873 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-envoy\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.233091 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.232948 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-credential-socket\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.233091 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.233006 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-socket\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.233218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.233091 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj2rv\" (UniqueName: \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-kube-api-access-wj2rv\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.233218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.233135 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-data\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.233218 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.233177 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5dd7cdf-8f73-48dc-a25e-507296833d93-istiod-ca-cert\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.233379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.233244 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-podinfo\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.233379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.233283 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-token\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.233379 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.233325 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-certs\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.334295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.334249 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-podinfo\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.334626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.334598 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-token\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.334775 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.334653 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-certs\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.334775 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.334722 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-envoy\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.334916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.334775 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-credential-socket\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.334916 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.334832 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-socket\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.335028 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.334920 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj2rv\" (UniqueName: \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-kube-api-access-wj2rv\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.335028 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.334968 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-data\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.335028 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.334991 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5dd7cdf-8f73-48dc-a25e-507296833d93-istiod-ca-cert\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.335280 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.335256 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-certs\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.335561 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.335527 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-socket\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.335812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.335788 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5dd7cdf-8f73-48dc-a25e-507296833d93-istiod-ca-cert\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.336310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.336135 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-data\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.336310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.336233 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-credential-socket\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.338160 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.338119 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-envoy\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.338909 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.338885 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-podinfo\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.390590 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.390556 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-token\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.391226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.391104 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj2rv\" (UniqueName: \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-kube-api-access-wj2rv\") pod \"e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.417030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.416960 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:01.851945 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:51:01.851910 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dd7cdf_8f73_48dc_a25e_507296833d93.slice/crio-76d71118df91742576b3a07e02447c337714eb07c868255a968d3cb7ba02c488 WatchSource:0}: Error finding container 76d71118df91742576b3a07e02447c337714eb07c868255a968d3cb7ba02c488: Status 404 returned error can't find the container with id 76d71118df91742576b3a07e02447c337714eb07c868255a968d3cb7ba02c488 Nov 02 23:51:01.852216 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.852180 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd"] Nov 02 23:51:01.855382 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.855341 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:51:01.855506 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.855429 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:51:01.855506 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:01.855470 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:51:02.002783 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:02.002106 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" event={"ID":"e5dd7cdf-8f73-48dc-a25e-507296833d93","Type":"ContainerStarted","Data":"900c6b114b6e4d1976350fe0429c3052e5488714502315978c3c297af32ba5e5"} Nov 02 23:51:02.002783 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:02.002194 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" event={"ID":"e5dd7cdf-8f73-48dc-a25e-507296833d93","Type":"ContainerStarted","Data":"76d71118df91742576b3a07e02447c337714eb07c868255a968d3cb7ba02c488"} Nov 02 23:51:02.113179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:02.113045 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" podStartSLOduration=1.113027914 podStartE2EDuration="1.113027914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:51:02.11121793 +0000 UTC m=+3455.518529759" watchObservedRunningTime="2026-11-02 23:51:02.113027914 +0000 UTC m=+3455.520339745" Nov 02 23:51:02.419232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:02.417301 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:02.419232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:02.418882 5672 patch_prober.go:29] interesting pod/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 02 23:51:02.419232 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:02.418936 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 02 23:51:03.422941 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:03.422910 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:04.015802 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:04.015768 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:04.017585 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:04.017552 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:51:08.776093 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:08.776057 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j"] Nov 02 23:51:08.780515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:08.780486 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:51:08.834295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:08.834267 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j"] Nov 02 23:51:08.875723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:08.875696 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f2df33-dockercfg-nld9j\"" Nov 02 23:51:08.909095 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:08.909069 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f2df33\"" Nov 02 23:51:08.932109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:08.932081 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ac7709-9676-4bed-97ea-6ed764c76f86-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j\" (UID: \"35ac7709-9676-4bed-97ea-6ed764c76f86\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:51:08.932263 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:08.932170 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghrj\" (UniqueName: \"kubernetes.io/projected/35ac7709-9676-4bed-97ea-6ed764c76f86-kube-api-access-dghrj\") pod \"payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j\" (UID: \"35ac7709-9676-4bed-97ea-6ed764c76f86\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:51:09.033580 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:09.033501 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ac7709-9676-4bed-97ea-6ed764c76f86-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j\" (UID: \"35ac7709-9676-4bed-97ea-6ed764c76f86\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:51:09.033580 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:09.033576 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghrj\" (UniqueName: \"kubernetes.io/projected/35ac7709-9676-4bed-97ea-6ed764c76f86-kube-api-access-dghrj\") pod \"payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j\" (UID: \"35ac7709-9676-4bed-97ea-6ed764c76f86\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:51:09.034237 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:09.034216 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ac7709-9676-4bed-97ea-6ed764c76f86-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j\" (UID: \"35ac7709-9676-4bed-97ea-6ed764c76f86\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:51:09.086253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:09.086219 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghrj\" (UniqueName: \"kubernetes.io/projected/35ac7709-9676-4bed-97ea-6ed764c76f86-kube-api-access-dghrj\") pod \"payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j\" (UID: \"35ac7709-9676-4bed-97ea-6ed764c76f86\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:51:09.095113 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:09.095085 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:51:09.462272 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:09.462243 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j"] Nov 02 23:51:09.464297 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:51:09.464265 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ac7709_9676_4bed_97ea_6ed764c76f86.slice/crio-f69ad479fcc5ad04f625dbc62496490bf7804f71be3e6b1990d9325296da2598 WatchSource:0}: Error finding container f69ad479fcc5ad04f625dbc62496490bf7804f71be3e6b1990d9325296da2598: Status 404 returned error can't find the container with id f69ad479fcc5ad04f625dbc62496490bf7804f71be3e6b1990d9325296da2598 Nov 02 23:51:10.057385 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:10.057347 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" event={"ID":"35ac7709-9676-4bed-97ea-6ed764c76f86","Type":"ContainerStarted","Data":"fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003"} Nov 02 23:51:10.057892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:10.057411 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" event={"ID":"35ac7709-9676-4bed-97ea-6ed764c76f86","Type":"ContainerStarted","Data":"f69ad479fcc5ad04f625dbc62496490bf7804f71be3e6b1990d9325296da2598"} Nov 02 23:51:10.057892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:10.057582 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:51:10.163933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:10.163880 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" podStartSLOduration=2.163866522 podStartE2EDuration="2.163866522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:51:10.163282727 +0000 UTC m=+3463.570594558" watchObservedRunningTime="2026-11-02 23:51:10.163866522 +0000 UTC m=+3463.571178350" Nov 02 23:51:21.067493 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:21.067451 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:51:26.020734 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:51:26.020698 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:51:44.236725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.236685 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n"] Nov 02 23:51:44.241994 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.241970 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:51:44.322376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.322344 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hvxl\" (UniqueName: \"kubernetes.io/projected/54892835-e11d-4301-9977-957d2ed95e89-kube-api-access-4hvxl\") pod \"payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n\" (UID: \"54892835-e11d-4301-9977-957d2ed95e89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:51:44.322567 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.322477 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54892835-e11d-4301-9977-957d2ed95e89-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n\" (UID: \"54892835-e11d-4301-9977-957d2ed95e89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:51:44.336929 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.336899 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c11b50\"" Nov 02 23:51:44.337134 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.336978 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c11b50-dockercfg-xv5m4\"" Nov 02 23:51:44.341455 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.341411 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n"] Nov 02 23:51:44.423859 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.423817 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54892835-e11d-4301-9977-957d2ed95e89-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n\" (UID: \"54892835-e11d-4301-9977-957d2ed95e89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:51:44.424051 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.423995 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hvxl\" (UniqueName: \"kubernetes.io/projected/54892835-e11d-4301-9977-957d2ed95e89-kube-api-access-4hvxl\") pod \"payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n\" (UID: \"54892835-e11d-4301-9977-957d2ed95e89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:51:44.425194 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.425125 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54892835-e11d-4301-9977-957d2ed95e89-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n\" (UID: \"54892835-e11d-4301-9977-957d2ed95e89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:51:44.482920 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.482884 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hvxl\" (UniqueName: \"kubernetes.io/projected/54892835-e11d-4301-9977-957d2ed95e89-kube-api-access-4hvxl\") pod \"payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n\" (UID: \"54892835-e11d-4301-9977-957d2ed95e89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:51:44.559164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.559065 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:51:44.927059 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:44.927025 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n"] Nov 02 23:51:44.928916 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:51:44.928884 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54892835_e11d_4301_9977_957d2ed95e89.slice/crio-9b6470d31578b1d6f568a7584a376186b5938fa010be37a96606c4829a431e95 WatchSource:0}: Error finding container 9b6470d31578b1d6f568a7584a376186b5938fa010be37a96606c4829a431e95: Status 404 returned error can't find the container with id 9b6470d31578b1d6f568a7584a376186b5938fa010be37a96606c4829a431e95 Nov 02 23:51:45.291076 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:45.291045 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" event={"ID":"54892835-e11d-4301-9977-957d2ed95e89","Type":"ContainerStarted","Data":"11ce77f668fff1b2e50f929ce172b13a96d1102cba5f843e5cc592f1f573eaa8"} Nov 02 23:51:45.291490 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:45.291093 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" event={"ID":"54892835-e11d-4301-9977-957d2ed95e89","Type":"ContainerStarted","Data":"9b6470d31578b1d6f568a7584a376186b5938fa010be37a96606c4829a431e95"} Nov 02 23:51:45.291490 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:45.291197 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:51:45.397102 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:45.397044 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" podStartSLOduration=1.3970287940000001 podStartE2EDuration="1.397028794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:51:45.396310391 +0000 UTC m=+3498.803622220" watchObservedRunningTime="2026-11-02 23:51:45.397028794 +0000 UTC m=+3498.804340623" Nov 02 23:51:48.309274 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:48.309241 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:51:56.020529 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:51:56.020493 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:51:56.300318 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:51:56.300241 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:52:10.416471 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.416434 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z"] Nov 02 23:52:10.420497 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.420473 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:52:10.439223 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.439192 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z\" (UID: \"ff85da83-99f1-472c-9c28-2e2f7a1f15ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:52:10.439371 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.439230 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgzt\" (UniqueName: \"kubernetes.io/projected/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-kube-api-access-cbgzt\") pod \"payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z\" (UID: \"ff85da83-99f1-472c-9c28-2e2f7a1f15ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:52:10.473584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.473554 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z"] Nov 02 23:52:10.516485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.516454 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-661a20\"" Nov 02 23:52:10.516668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.516550 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-661a20-dockercfg-6wn2v\"" Nov 02 23:52:10.540656 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.540616 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgzt\" (UniqueName: \"kubernetes.io/projected/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-kube-api-access-cbgzt\") pod \"payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z\" (UID: \"ff85da83-99f1-472c-9c28-2e2f7a1f15ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:52:10.540960 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.540856 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z\" (UID: \"ff85da83-99f1-472c-9c28-2e2f7a1f15ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:52:10.541737 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.541711 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z\" (UID: \"ff85da83-99f1-472c-9c28-2e2f7a1f15ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:52:10.595310 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.595277 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgzt\" (UniqueName: \"kubernetes.io/projected/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-kube-api-access-cbgzt\") pod \"payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z\" (UID: \"ff85da83-99f1-472c-9c28-2e2f7a1f15ff\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:52:10.735544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:10.735515 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:52:11.114500 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:11.114468 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z"] Nov 02 23:52:11.116213 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:52:11.116187 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff85da83_99f1_472c_9c28_2e2f7a1f15ff.slice/crio-6018a2fdeaa89a619fd17d031e55212056c57ddc7f803a7389e78660617623f0 WatchSource:0}: Error finding container 6018a2fdeaa89a619fd17d031e55212056c57ddc7f803a7389e78660617623f0: Status 404 returned error can't find the container with id 6018a2fdeaa89a619fd17d031e55212056c57ddc7f803a7389e78660617623f0 Nov 02 23:52:11.474949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:11.474913 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" event={"ID":"ff85da83-99f1-472c-9c28-2e2f7a1f15ff","Type":"ContainerStarted","Data":"42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77"} Nov 02 23:52:11.475445 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:11.474976 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" event={"ID":"ff85da83-99f1-472c-9c28-2e2f7a1f15ff","Type":"ContainerStarted","Data":"6018a2fdeaa89a619fd17d031e55212056c57ddc7f803a7389e78660617623f0"} Nov 02 23:52:11.475445 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:11.475005 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:52:11.585114 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:11.585047 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" podStartSLOduration=1.585029611 podStartE2EDuration="1.585029611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:11.580529939 +0000 UTC m=+3524.987841771" watchObservedRunningTime="2026-11-02 23:52:11.585029611 +0000 UTC m=+3524.992341443" Nov 02 23:52:22.485644 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:22.485613 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:52:25.878160 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.876105 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t"] Nov 02 23:52:25.881018 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.880998 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:25.911480 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.911453 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-credential-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:25.911647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.911509 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-token\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:25.911647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.911537 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:25.911647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.911621 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-certs\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:25.911823 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.911648 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3cff8c2-f122-4b13-a292-31bf15b6098b-istiod-ca-cert\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:25.911823 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.911671 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-envoy\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:25.911823 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.911688 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-podinfo\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:25.911823 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.911805 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8vc\" (UniqueName: \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-kube-api-access-bw8vc\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:25.912030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.911834 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-data\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:25.934456 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.934426 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t"] Nov 02 23:52:25.972580 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:25.972553 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e65845-openshift-default-dockercfg-fgqjd\"" Nov 02 23:52:26.013447 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.013409 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-certs\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.013646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.013479 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3cff8c2-f122-4b13-a292-31bf15b6098b-istiod-ca-cert\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.013646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.013515 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-envoy\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.013646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.013533 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-podinfo\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.013646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.013572 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8vc\" (UniqueName: \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-kube-api-access-bw8vc\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.013646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.013591 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-data\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.013921 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.013676 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-credential-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.013921 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.013747 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-token\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.013921 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.013875 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.014272 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.014255 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.014968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.014543 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-certs\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.014968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.014740 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-credential-socket\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.014968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.014799 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-data\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.014968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.014894 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3cff8c2-f122-4b13-a292-31bf15b6098b-istiod-ca-cert\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.016634 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.016612 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-podinfo\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.016814 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.016796 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-envoy\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.023887 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:52:26.023857 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:52:26.067710 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.067669 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-token\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.067907 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.067883 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8vc\" (UniqueName: \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-kube-api-access-bw8vc\") pod \"e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.195358 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.195289 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:26.613211 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.613183 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t"] Nov 02 23:52:26.614674 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:52:26.614646 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cff8c2_f122_4b13_a292_31bf15b6098b.slice/crio-cec7cff5c6c2e1341874b42952b7d5f60b0eebc935864c6d643e87ae39040768 WatchSource:0}: Error finding container cec7cff5c6c2e1341874b42952b7d5f60b0eebc935864c6d643e87ae39040768: Status 404 returned error can't find the container with id cec7cff5c6c2e1341874b42952b7d5f60b0eebc935864c6d643e87ae39040768 Nov 02 23:52:26.617490 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.617458 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:52:26.617604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.617534 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:52:26.617674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:26.617602 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:52:27.590176 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:27.590124 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" event={"ID":"a3cff8c2-f122-4b13-a292-31bf15b6098b","Type":"ContainerStarted","Data":"ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754"} Nov 02 23:52:27.590667 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:27.590229 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" event={"ID":"a3cff8c2-f122-4b13-a292-31bf15b6098b","Type":"ContainerStarted","Data":"cec7cff5c6c2e1341874b42952b7d5f60b0eebc935864c6d643e87ae39040768"} Nov 02 23:52:27.758131 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:27.758049 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" podStartSLOduration=2.758029992 podStartE2EDuration="2.758029992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:27.756948576 +0000 UTC m=+3541.164260407" watchObservedRunningTime="2026-11-02 23:52:27.758029992 +0000 UTC m=+3541.165341824" Nov 02 23:52:28.195971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:28.195933 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:28.197462 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:28.197435 5672 patch_prober.go:29] interesting pod/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 02 23:52:28.197604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:28.197501 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" podUID="a3cff8c2-f122-4b13-a292-31bf15b6098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 02 23:52:29.050170 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:29.049283 5672 scope.go:122] "RemoveContainer" containerID="612d3a1dd0bcbdef2d248d1c57e5aae6e81f9b15734c2b6cc05d83f5b82b049b" Nov 02 23:52:29.196251 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:29.196211 5672 patch_prober.go:29] interesting pod/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 02 23:52:29.196429 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:29.196287 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" podUID="a3cff8c2-f122-4b13-a292-31bf15b6098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 02 23:52:30.195749 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:30.195713 5672 patch_prober.go:29] interesting pod/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 02 23:52:30.196282 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:30.195767 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" podUID="a3cff8c2-f122-4b13-a292-31bf15b6098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 02 23:52:31.195674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.195640 5672 patch_prober.go:29] interesting pod/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 02 23:52:31.195907 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.195712 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" podUID="a3cff8c2-f122-4b13-a292-31bf15b6098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 02 23:52:31.355058 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.355023 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d"] Nov 02 23:52:31.360430 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.360408 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:52:31.411263 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.411217 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d"] Nov 02 23:52:31.457082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.456997 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4b63d8-dockercfg-vjb8n\"" Nov 02 23:52:31.457347 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.457237 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4b63d8\"" Nov 02 23:52:31.509032 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.508996 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48fbp\" (UniqueName: \"kubernetes.io/projected/ec024bb7-0d87-4472-9cd9-4be146118f14-kube-api-access-48fbp\") pod \"payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d\" (UID: \"ec024bb7-0d87-4472-9cd9-4be146118f14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:52:31.509589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.509399 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec024bb7-0d87-4472-9cd9-4be146118f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d\" (UID: \"ec024bb7-0d87-4472-9cd9-4be146118f14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:52:31.517208 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.515491 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm"] Nov 02 23:52:31.519576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.519555 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:52:31.576852 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.575126 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm"] Nov 02 23:52:31.611034 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.610995 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c2e004-7a8a-431d-8380-b96ba2f0a628-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm\" (UID: \"e7c2e004-7a8a-431d-8380-b96ba2f0a628\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:52:31.611234 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.611167 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48fbp\" (UniqueName: \"kubernetes.io/projected/ec024bb7-0d87-4472-9cd9-4be146118f14-kube-api-access-48fbp\") pod \"payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d\" (UID: \"ec024bb7-0d87-4472-9cd9-4be146118f14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:52:31.611315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.611254 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxn99\" (UniqueName: \"kubernetes.io/projected/e7c2e004-7a8a-431d-8380-b96ba2f0a628-kube-api-access-pxn99\") pod \"payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm\" (UID: \"e7c2e004-7a8a-431d-8380-b96ba2f0a628\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:52:31.611315 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.611280 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec024bb7-0d87-4472-9cd9-4be146118f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d\" (UID: \"ec024bb7-0d87-4472-9cd9-4be146118f14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:52:31.612013 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.611996 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec024bb7-0d87-4472-9cd9-4be146118f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d\" (UID: \"ec024bb7-0d87-4472-9cd9-4be146118f14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:52:31.664510 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.664482 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48fbp\" (UniqueName: \"kubernetes.io/projected/ec024bb7-0d87-4472-9cd9-4be146118f14-kube-api-access-48fbp\") pod \"payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d\" (UID: \"ec024bb7-0d87-4472-9cd9-4be146118f14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:52:31.674359 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.674328 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:52:31.713041 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.712972 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxn99\" (UniqueName: \"kubernetes.io/projected/e7c2e004-7a8a-431d-8380-b96ba2f0a628-kube-api-access-pxn99\") pod \"payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm\" (UID: \"e7c2e004-7a8a-431d-8380-b96ba2f0a628\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:52:31.713190 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.713086 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c2e004-7a8a-431d-8380-b96ba2f0a628-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm\" (UID: \"e7c2e004-7a8a-431d-8380-b96ba2f0a628\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:52:31.713773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.713745 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c2e004-7a8a-431d-8380-b96ba2f0a628-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm\" (UID: \"e7c2e004-7a8a-431d-8380-b96ba2f0a628\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:52:31.769723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.769687 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxn99\" (UniqueName: \"kubernetes.io/projected/e7c2e004-7a8a-431d-8380-b96ba2f0a628-kube-api-access-pxn99\") pod \"payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm\" (UID: \"e7c2e004-7a8a-431d-8380-b96ba2f0a628\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:52:31.860021 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:31.859987 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:52:32.082826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:32.082791 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d"] Nov 02 23:52:32.083865 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:52:32.083812 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec024bb7_0d87_4472_9cd9_4be146118f14.slice/crio-e021505c4b27ad89d8d9b3a4878356cec6055639a53858dc156f54e27e607981 WatchSource:0}: Error finding container e021505c4b27ad89d8d9b3a4878356cec6055639a53858dc156f54e27e607981: Status 404 returned error can't find the container with id e021505c4b27ad89d8d9b3a4878356cec6055639a53858dc156f54e27e607981 Nov 02 23:52:32.196556 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:32.196528 5672 patch_prober.go:29] interesting pod/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 02 23:52:32.196871 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:32.196572 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" podUID="a3cff8c2-f122-4b13-a292-31bf15b6098b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 02 23:52:32.262104 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:52:32.260533 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c2e004_7a8a_431d_8380_b96ba2f0a628.slice/crio-4417ad342dc5c620ea414cc89a8ab4e88e37693aa420d7933437b9370c3ed5ee WatchSource:0}: Error finding container 4417ad342dc5c620ea414cc89a8ab4e88e37693aa420d7933437b9370c3ed5ee: Status 404 returned error can't find the container with id 4417ad342dc5c620ea414cc89a8ab4e88e37693aa420d7933437b9370c3ed5ee Nov 02 23:52:32.262104 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:32.262010 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm"] Nov 02 23:52:32.645226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:32.633973 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" event={"ID":"ec024bb7-0d87-4472-9cd9-4be146118f14","Type":"ContainerStarted","Data":"52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd"} Nov 02 23:52:32.645226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:32.634039 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" event={"ID":"ec024bb7-0d87-4472-9cd9-4be146118f14","Type":"ContainerStarted","Data":"e021505c4b27ad89d8d9b3a4878356cec6055639a53858dc156f54e27e607981"} Nov 02 23:52:32.645226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:32.634936 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:52:32.651359 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:32.651328 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" event={"ID":"e7c2e004-7a8a-431d-8380-b96ba2f0a628","Type":"ContainerStarted","Data":"ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79"} Nov 02 23:52:32.651510 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:32.651382 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" event={"ID":"e7c2e004-7a8a-431d-8380-b96ba2f0a628","Type":"ContainerStarted","Data":"4417ad342dc5c620ea414cc89a8ab4e88e37693aa420d7933437b9370c3ed5ee"} Nov 02 23:52:32.741432 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:32.741371 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" podStartSLOduration=1.741357211 podStartE2EDuration="1.741357211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:32.740036022 +0000 UTC m=+3546.147347852" watchObservedRunningTime="2026-11-02 23:52:32.741357211 +0000 UTC m=+3546.148669068" Nov 02 23:52:32.847190 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:32.847106 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" podStartSLOduration=1.847091449 podStartE2EDuration="1.847091449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:32.844858233 +0000 UTC m=+3546.252170062" watchObservedRunningTime="2026-11-02 23:52:32.847091449 +0000 UTC m=+3546.254403287" Nov 02 23:52:33.201433 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:33.201401 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:33.201862 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:33.201838 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:33.202722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:33.202698 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:52:33.659398 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:33.659352 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:52:44.045880 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:44.045841 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c"] Nov 02 23:52:44.090459 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:44.090431 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-005fbc5e-zjw5c"] Nov 02 23:52:44.671722 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:44.671692 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:52:44.672166 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:44.672134 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:52:44.713067 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:44.713035 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6be90a6-8029-446b-a4dd-94080cfccb29" path="/var/lib/kubelet/pods/f6be90a6-8029-446b-a4dd-94080cfccb29/volumes" Nov 02 23:52:49.235728 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:49.235689 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:52:55.051716 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:55.051682 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss"] Nov 02 23:52:55.090831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:55.090801 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7142b1e5-2dsss"] Nov 02 23:52:56.023464 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:52:56.023421 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:52:56.714019 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:56.713982 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72800408-6af7-4109-8ddc-277f0511a589" path="/var/lib/kubelet/pods/72800408-6af7-4109-8ddc-277f0511a589/volumes" Nov 02 23:52:59.835335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:59.835295 5672 generic.go:356] "Generic (PLEG): container finished" podID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerID="900c6b114b6e4d1976350fe0429c3052e5488714502315978c3c297af32ba5e5" exitCode=0 Nov 02 23:52:59.835793 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:59.835368 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" event={"ID":"e5dd7cdf-8f73-48dc-a25e-507296833d93","Type":"ContainerDied","Data":"900c6b114b6e4d1976350fe0429c3052e5488714502315978c3c297af32ba5e5"} Nov 02 23:52:59.835845 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:59.835814 5672 scope.go:122] "RemoveContainer" containerID="900c6b114b6e4d1976350fe0429c3052e5488714502315978c3c297af32ba5e5" Nov 02 23:52:59.837409 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:59.837379 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:52:59.837515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:59.837450 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:52:59.837515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:52:59.837482 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:53:00.859171 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:00.859093 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" event={"ID":"e5dd7cdf-8f73-48dc-a25e-507296833d93","Type":"ContainerStarted","Data":"e4c0d55570e03210ca0add0e9e6709853b6d7c1307e605050b6a6e255dd360b4"} Nov 02 23:53:01.417305 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:01.417270 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:53:01.418703 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:01.418673 5672 patch_prober.go:29] interesting pod/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 02 23:53:01.418862 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:01.418725 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 02 23:53:02.418399 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:02.418332 5672 patch_prober.go:29] interesting pod/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 02 23:53:02.418817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:02.418421 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 02 23:53:03.417331 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:03.417296 5672 patch_prober.go:29] interesting pod/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 02 23:53:03.417534 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:03.417355 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 02 23:53:04.016224 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:04.016191 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:53:04.418182 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:04.418078 5672 patch_prober.go:29] interesting pod/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 02 23:53:04.418347 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:04.418189 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 02 23:53:05.417772 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:05.417734 5672 patch_prober.go:29] interesting pod/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 02 23:53:05.418229 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:05.417795 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 02 23:53:06.422931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:06.422901 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:53:06.424731 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:06.424704 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:53:26.021966 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:53:26.021927 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:53:29.211359 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:29.211319 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:53:32.017869 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.017834 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh"] Nov 02 23:53:32.023272 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.023249 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:53:32.065014 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.064977 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7hz\" (UniqueName: \"kubernetes.io/projected/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-kube-api-access-4v7hz\") pod \"payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh\" (UID: \"69f43db0-2fee-4fb8-bc4d-dd8aee83e903\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:53:32.065214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.065058 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh\" (UID: \"69f43db0-2fee-4fb8-bc4d-dd8aee83e903\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:53:32.074816 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.074788 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh"] Nov 02 23:53:32.117635 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.117513 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a8a1dd\"" Nov 02 23:53:32.117896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.117595 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a8a1dd-dockercfg-27mpc\"" Nov 02 23:53:32.166236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.166194 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7hz\" (UniqueName: \"kubernetes.io/projected/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-kube-api-access-4v7hz\") pod \"payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh\" (UID: \"69f43db0-2fee-4fb8-bc4d-dd8aee83e903\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:53:32.166470 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.166302 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh\" (UID: \"69f43db0-2fee-4fb8-bc4d-dd8aee83e903\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:53:32.167508 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.167479 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh\" (UID: \"69f43db0-2fee-4fb8-bc4d-dd8aee83e903\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:53:32.190047 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.190011 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd"] Nov 02 23:53:32.194124 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.194105 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:53:32.219026 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.219002 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7hz\" (UniqueName: \"kubernetes.io/projected/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-kube-api-access-4v7hz\") pod \"payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh\" (UID: \"69f43db0-2fee-4fb8-bc4d-dd8aee83e903\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:53:32.247350 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.247299 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd"] Nov 02 23:53:32.268258 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.267911 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtbz\" (UniqueName: \"kubernetes.io/projected/d32a755f-2f63-4522-b2d0-084792a5afbe-kube-api-access-vvtbz\") pod \"payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd\" (UID: \"d32a755f-2f63-4522-b2d0-084792a5afbe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:53:32.268258 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.268173 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32a755f-2f63-4522-b2d0-084792a5afbe-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd\" (UID: \"d32a755f-2f63-4522-b2d0-084792a5afbe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:53:32.337544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.337513 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:53:32.369665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.369627 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32a755f-2f63-4522-b2d0-084792a5afbe-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd\" (UID: \"d32a755f-2f63-4522-b2d0-084792a5afbe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:53:32.369830 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.369687 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtbz\" (UniqueName: \"kubernetes.io/projected/d32a755f-2f63-4522-b2d0-084792a5afbe-kube-api-access-vvtbz\") pod \"payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd\" (UID: \"d32a755f-2f63-4522-b2d0-084792a5afbe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:53:32.370375 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.370352 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32a755f-2f63-4522-b2d0-084792a5afbe-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd\" (UID: \"d32a755f-2f63-4522-b2d0-084792a5afbe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:53:32.425071 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.423963 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtbz\" (UniqueName: \"kubernetes.io/projected/d32a755f-2f63-4522-b2d0-084792a5afbe-kube-api-access-vvtbz\") pod \"payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd\" (UID: \"d32a755f-2f63-4522-b2d0-084792a5afbe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:53:32.507503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.507462 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:53:32.703324 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.703296 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh"] Nov 02 23:53:32.705089 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:53:32.705049 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f43db0_2fee_4fb8_bc4d_dd8aee83e903.slice/crio-1f1dd7c86b4aa35949a9659db66145cc11955618ec4f19b700ab5bb4962f486f WatchSource:0}: Error finding container 1f1dd7c86b4aa35949a9659db66145cc11955618ec4f19b700ab5bb4962f486f: Status 404 returned error can't find the container with id 1f1dd7c86b4aa35949a9659db66145cc11955618ec4f19b700ab5bb4962f486f Nov 02 23:53:32.877006 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:32.876975 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd"] Nov 02 23:53:32.878862 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:53:32.878831 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32a755f_2f63_4522_b2d0_084792a5afbe.slice/crio-75e67bc5e049b1c6c67b0d3c1175c5b45994ed2f199499ee8ddfc916b2486ab1 WatchSource:0}: Error finding container 75e67bc5e049b1c6c67b0d3c1175c5b45994ed2f199499ee8ddfc916b2486ab1: Status 404 returned error can't find the container with id 75e67bc5e049b1c6c67b0d3c1175c5b45994ed2f199499ee8ddfc916b2486ab1 Nov 02 23:53:33.097671 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:33.097572 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" event={"ID":"d32a755f-2f63-4522-b2d0-084792a5afbe","Type":"ContainerStarted","Data":"e1e26283b0edd5084d738e50f08772635ef11e0984b8565ab0c459ad534a6936"} Nov 02 23:53:33.097671 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:33.097631 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" event={"ID":"d32a755f-2f63-4522-b2d0-084792a5afbe","Type":"ContainerStarted","Data":"75e67bc5e049b1c6c67b0d3c1175c5b45994ed2f199499ee8ddfc916b2486ab1"} Nov 02 23:53:33.098249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:33.097673 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:53:33.099662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:33.099632 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" event={"ID":"69f43db0-2fee-4fb8-bc4d-dd8aee83e903","Type":"ContainerStarted","Data":"c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba"} Nov 02 23:53:33.099812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:33.099680 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" event={"ID":"69f43db0-2fee-4fb8-bc4d-dd8aee83e903","Type":"ContainerStarted","Data":"1f1dd7c86b4aa35949a9659db66145cc11955618ec4f19b700ab5bb4962f486f"} Nov 02 23:53:33.099812 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:33.099791 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:53:33.203709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:33.203641 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" podStartSLOduration=1.203621202 podStartE2EDuration="1.203621202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:53:33.202417083 +0000 UTC m=+3606.609728913" watchObservedRunningTime="2026-11-02 23:53:33.203621202 +0000 UTC m=+3606.610933034" Nov 02 23:53:33.309658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:33.309593 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" podStartSLOduration=2.309574206 podStartE2EDuration="2.309574206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:53:33.307504405 +0000 UTC m=+3606.714816248" watchObservedRunningTime="2026-11-02 23:53:33.309574206 +0000 UTC m=+3606.716886037" Nov 02 23:53:44.110489 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:44.110458 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:53:44.110937 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:53:44.110527 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:53:56.020598 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:53:56.020560 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:54:16.656224 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:16.656185 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j"] Nov 02 23:54:21.657348 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:21.657305 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" podUID="35ac7709-9676-4bed-97ea-6ed764c76f86" containerName="payload-processing" containerID="cri-o://fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003" gracePeriod=25 Nov 02 23:54:21.928090 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:21.928065 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:54:21.962747 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:21.962716 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35ac7709-9676-4bed-97ea-6ed764c76f86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ac7709-9676-4bed-97ea-6ed764c76f86-plugins-config-volume\") pod \"35ac7709-9676-4bed-97ea-6ed764c76f86\" (UID: \"35ac7709-9676-4bed-97ea-6ed764c76f86\") " Nov 02 23:54:21.963143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:21.963116 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ac7709-9676-4bed-97ea-6ed764c76f86-plugins-config-volume" pod "35ac7709-9676-4bed-97ea-6ed764c76f86" (UID: "35ac7709-9676-4bed-97ea-6ed764c76f86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:22.063487 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.063446 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ac7709-9676-4bed-97ea-6ed764c76f86-kube-api-access-dghrj\" (UniqueName: \"kubernetes.io/projected/35ac7709-9676-4bed-97ea-6ed764c76f86-kube-api-access-dghrj\") pod \"35ac7709-9676-4bed-97ea-6ed764c76f86\" (UID: \"35ac7709-9676-4bed-97ea-6ed764c76f86\") " Nov 02 23:54:22.064059 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.064037 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ac7709-9676-4bed-97ea-6ed764c76f86-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:54:22.065686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.065657 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ac7709-9676-4bed-97ea-6ed764c76f86-kube-api-access-dghrj" pod "35ac7709-9676-4bed-97ea-6ed764c76f86" (UID: "35ac7709-9676-4bed-97ea-6ed764c76f86"). InnerVolumeSpecName "kube-api-access-dghrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:22.165797 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.165718 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dghrj\" (UniqueName: \"kubernetes.io/projected/35ac7709-9676-4bed-97ea-6ed764c76f86-kube-api-access-dghrj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:54:22.454733 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.454641 5672 generic.go:356] "Generic (PLEG): container finished" podID="35ac7709-9676-4bed-97ea-6ed764c76f86" containerID="fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003" exitCode=0 Nov 02 23:54:22.454733 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.454693 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" Nov 02 23:54:22.454986 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.454687 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" event={"ID":"35ac7709-9676-4bed-97ea-6ed764c76f86","Type":"ContainerDied","Data":"fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003"} Nov 02 23:54:22.454986 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.454832 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j" event={"ID":"35ac7709-9676-4bed-97ea-6ed764c76f86","Type":"ContainerDied","Data":"f69ad479fcc5ad04f625dbc62496490bf7804f71be3e6b1990d9325296da2598"} Nov 02 23:54:22.454986 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.454863 5672 scope.go:122] "RemoveContainer" containerID="fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003" Nov 02 23:54:22.467542 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.467521 5672 scope.go:122] "RemoveContainer" containerID="fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003" Nov 02 23:54:22.467848 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:54:22.467826 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003\": container with ID starting with fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003 not found: ID does not exist" containerID="fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003" Nov 02 23:54:22.467942 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.467858 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003"} err="failed to get container status \"fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003\": rpc error: code = NotFound desc = could not find container \"fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003\": container with ID starting with fb265a44595d58d0778878877678c6a45f01a97d5bd3ed5b045586e80a3ca003 not found: ID does not exist" Nov 02 23:54:22.608548 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.608510 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j"] Nov 02 23:54:22.612110 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.612083 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f2df33-7f594579f-zxw7j"] Nov 02 23:54:22.713881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:22.713809 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35ac7709-9676-4bed-97ea-6ed764c76f86" path="/var/lib/kubelet/pods/35ac7709-9676-4bed-97ea-6ed764c76f86/volumes" Nov 02 23:54:24.504883 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:24.504850 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:54:26.020909 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:54:26.020870 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:54:26.044945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:26.044914 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b"] Nov 02 23:54:26.091397 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:26.091359 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03bc356b-c2t9b"] Nov 02 23:54:26.713912 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:26.713881 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f" path="/var/lib/kubelet/pods/e5bafe2a-66b3-4e3b-a1dd-dacc2f72ed2f/volumes" Nov 02 23:54:29.097783 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:29.097744 5672 scope.go:122] "RemoveContainer" containerID="cd16b4c24e030b9cbaf6004f0b180b5f2449998c74d034bb32048666a1c50fad" Nov 02 23:54:29.108544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:29.108512 5672 scope.go:122] "RemoveContainer" containerID="6a4f162be64f211f0d9c737f636ac6a2100528d073e780d1b6ff11d8c5b453e3" Nov 02 23:54:37.048485 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:37.048445 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465"] Nov 02 23:54:37.052906 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:37.052873 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1598c534-hc465"] Nov 02 23:54:38.712640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:38.712608 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a894456-5e05-4f39-86e2-54ce5e24f679" path="/var/lib/kubelet/pods/4a894456-5e05-4f39-86e2-54ce5e24f679/volumes" Nov 02 23:54:49.432825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:49.432791 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n"] Nov 02 23:54:54.434088 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.434040 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" podUID="54892835-e11d-4301-9977-957d2ed95e89" containerName="payload-pre-processing" containerID="cri-o://11ce77f668fff1b2e50f929ce172b13a96d1102cba5f843e5cc592f1f573eaa8" gracePeriod=25 Nov 02 23:54:54.692753 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.692691 5672 generic.go:356] "Generic (PLEG): container finished" podID="54892835-e11d-4301-9977-957d2ed95e89" containerID="11ce77f668fff1b2e50f929ce172b13a96d1102cba5f843e5cc592f1f573eaa8" exitCode=0 Nov 02 23:54:54.692879 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.692770 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" event={"ID":"54892835-e11d-4301-9977-957d2ed95e89","Type":"ContainerDied","Data":"11ce77f668fff1b2e50f929ce172b13a96d1102cba5f843e5cc592f1f573eaa8"} Nov 02 23:54:54.692879 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.692835 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" event={"ID":"54892835-e11d-4301-9977-957d2ed95e89","Type":"ContainerDied","Data":"9b6470d31578b1d6f568a7584a376186b5938fa010be37a96606c4829a431e95"} Nov 02 23:54:54.692879 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.692859 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b6470d31578b1d6f568a7584a376186b5938fa010be37a96606c4829a431e95" Nov 02 23:54:54.703743 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.703721 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:54:54.875235 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.875196 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54892835-e11d-4301-9977-957d2ed95e89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54892835-e11d-4301-9977-957d2ed95e89-plugins-config-volume\") pod \"54892835-e11d-4301-9977-957d2ed95e89\" (UID: \"54892835-e11d-4301-9977-957d2ed95e89\") " Nov 02 23:54:54.875496 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.875480 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54892835-e11d-4301-9977-957d2ed95e89-kube-api-access-4hvxl\" (UniqueName: \"kubernetes.io/projected/54892835-e11d-4301-9977-957d2ed95e89-kube-api-access-4hvxl\") pod \"54892835-e11d-4301-9977-957d2ed95e89\" (UID: \"54892835-e11d-4301-9977-957d2ed95e89\") " Nov 02 23:54:54.875653 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.875626 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54892835-e11d-4301-9977-957d2ed95e89-plugins-config-volume" pod "54892835-e11d-4301-9977-957d2ed95e89" (UID: "54892835-e11d-4301-9977-957d2ed95e89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:54:54.875943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.875919 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54892835-e11d-4301-9977-957d2ed95e89-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:54:54.877817 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.877793 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54892835-e11d-4301-9977-957d2ed95e89-kube-api-access-4hvxl" pod "54892835-e11d-4301-9977-957d2ed95e89" (UID: "54892835-e11d-4301-9977-957d2ed95e89"). InnerVolumeSpecName "kube-api-access-4hvxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:54:54.976785 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:54.976750 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hvxl\" (UniqueName: \"kubernetes.io/projected/54892835-e11d-4301-9977-957d2ed95e89-kube-api-access-4hvxl\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:54:55.700226 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:55.700197 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n" Nov 02 23:54:55.856311 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:55.856271 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n"] Nov 02 23:54:55.901024 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:55.900976 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c11b50-645c4ddf67-5pb7n"] Nov 02 23:54:56.020955 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:54:56.020923 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:54:56.712701 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:54:56.712668 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54892835-e11d-4301-9977-957d2ed95e89" path="/var/lib/kubelet/pods/54892835-e11d-4301-9977-957d2ed95e89/volumes" Nov 02 23:55:01.043835 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:01.043797 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp"] Nov 02 23:55:01.090118 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:01.090078 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83c0e838-kkgwp"] Nov 02 23:55:02.712588 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:02.712548 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da40d92-aae0-46c7-b744-84a0a7575158" path="/var/lib/kubelet/pods/3da40d92-aae0-46c7-b744-84a0a7575158/volumes" Nov 02 23:55:12.045040 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:12.044955 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5"] Nov 02 23:55:12.089426 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:12.089392 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4696bedd-zsvn5"] Nov 02 23:55:12.713603 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:12.713569 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a302741c-5165-4dbe-ad80-cd06c96c54e8" path="/var/lib/kubelet/pods/a302741c-5165-4dbe-ad80-cd06c96c54e8/volumes" Nov 02 23:55:24.674021 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:24.673989 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:55:26.021108 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:55:26.021069 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:55:29.146791 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:29.146752 5672 scope.go:122] "RemoveContainer" containerID="c23b7a9c13e57ba71e87c29dd05e2edf787dfb825ced6d7ffee73f346ec60b86" Nov 02 23:55:36.047107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:36.047068 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr"] Nov 02 23:55:36.091396 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:36.091365 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d03e78-p6xmr"] Nov 02 23:55:36.713820 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:36.713788 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7193049d-1622-4e94-93f4-ac729c1d1d97" path="/var/lib/kubelet/pods/7193049d-1622-4e94-93f4-ac729c1d1d97/volumes" Nov 02 23:55:37.773924 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:37.773888 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r"] Nov 02 23:55:39.203911 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.203872 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq"] Nov 02 23:55:39.204507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.204486 5672 cpu_manager.go:414] "containerMap: removing container" podUID="54892835-e11d-4301-9977-957d2ed95e89" containerName="payload-pre-processing" Nov 02 23:55:39.204507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.204507 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54892835-e11d-4301-9977-957d2ed95e89" containerName="payload-pre-processing" Nov 02 23:55:39.204678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.204547 5672 cpu_manager.go:414] "containerMap: removing container" podUID="35ac7709-9676-4bed-97ea-6ed764c76f86" containerName="payload-processing" Nov 02 23:55:39.204678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.204552 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ac7709-9676-4bed-97ea-6ed764c76f86" containerName="payload-processing" Nov 02 23:55:39.204678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.204643 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ac7709-9676-4bed-97ea-6ed764c76f86" containerName="payload-processing" Nov 02 23:55:39.204678 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.204654 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="54892835-e11d-4301-9977-957d2ed95e89" containerName="payload-pre-processing" Nov 02 23:55:39.208416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.208399 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:39.260846 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.260811 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq"] Nov 02 23:55:39.303954 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.303903 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:55:39.304355 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.303927 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:55:39.305104 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.305080 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:39.305248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.305120 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnkhj\" (UniqueName: \"kubernetes.io/projected/dd444e9c-1e51-4137-9768-00e4ad65ff8b-kube-api-access-pnkhj\") pod \"maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:39.305248 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.305182 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:39.305381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.305267 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:55:39.305381 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.305295 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:55:39.406633 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.406600 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:39.406841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.406643 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnkhj\" (UniqueName: \"kubernetes.io/projected/dd444e9c-1e51-4137-9768-00e4ad65ff8b-kube-api-access-pnkhj\") pod \"maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:39.406841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.406691 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:39.407418 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.407393 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:39.409028 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.409007 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:39.459739 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.459670 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnkhj\" (UniqueName: \"kubernetes.io/projected/dd444e9c-1e51-4137-9768-00e4ad65ff8b-kube-api-access-pnkhj\") pod \"maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:39.521517 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.521489 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:39.885945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:39.885913 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq"] Nov 02 23:55:39.889014 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:55:39.888985 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd444e9c_1e51_4137_9768_00e4ad65ff8b.slice/crio-4b4c4b11a04b0226c54800f3aa5379fe0d1d1e34ab0152d4614b8923e65345b7 WatchSource:0}: Error finding container 4b4c4b11a04b0226c54800f3aa5379fe0d1d1e34ab0152d4614b8923e65345b7: Status 404 returned error can't find the container with id 4b4c4b11a04b0226c54800f3aa5379fe0d1d1e34ab0152d4614b8923e65345b7 Nov 02 23:55:40.020054 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:40.020017 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" event={"ID":"dd444e9c-1e51-4137-9768-00e4ad65ff8b","Type":"ContainerStarted","Data":"4b4c4b11a04b0226c54800f3aa5379fe0d1d1e34ab0152d4614b8923e65345b7"} Nov 02 23:55:41.029825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:41.029739 5672 generic.go:356] "Generic (PLEG): container finished" podID="dd444e9c-1e51-4137-9768-00e4ad65ff8b" containerID="3c7dd442b20338a6c3eb48b77aa1de66788f51e1bd4f89f957a5fc1b4faf62b3" exitCode=0 Nov 02 23:55:41.030195 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:41.029821 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" event={"ID":"dd444e9c-1e51-4137-9768-00e4ad65ff8b","Type":"ContainerDied","Data":"3c7dd442b20338a6c3eb48b77aa1de66788f51e1bd4f89f957a5fc1b4faf62b3"} Nov 02 23:55:42.185654 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:42.185632 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:42.239587 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:42.239556 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-openshift-ca\") pod \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " Nov 02 23:55:42.239748 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:42.239675 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd444e9c-1e51-4137-9768-00e4ad65ff8b-kube-api-access-pnkhj\" (UniqueName: \"kubernetes.io/projected/dd444e9c-1e51-4137-9768-00e4ad65ff8b-kube-api-access-pnkhj\") pod \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " Nov 02 23:55:42.239819 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:42.239773 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-xks-ca\") pod \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\" (UID: \"dd444e9c-1e51-4137-9768-00e4ad65ff8b\") " Nov 02 23:55:42.239905 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:42.239882 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-openshift-ca" pod "dd444e9c-1e51-4137-9768-00e4ad65ff8b" (UID: "dd444e9c-1e51-4137-9768-00e4ad65ff8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:55:42.240139 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:42.240121 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:55:42.241765 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:42.241743 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd444e9c-1e51-4137-9768-00e4ad65ff8b-kube-api-access-pnkhj" pod "dd444e9c-1e51-4137-9768-00e4ad65ff8b" (UID: "dd444e9c-1e51-4137-9768-00e4ad65ff8b"). InnerVolumeSpecName "kube-api-access-pnkhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:55:42.241842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:42.241789 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-xks-ca" pod "dd444e9c-1e51-4137-9768-00e4ad65ff8b" (UID: "dd444e9c-1e51-4137-9768-00e4ad65ff8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:55:42.341804 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:42.341722 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd444e9c-1e51-4137-9768-00e4ad65ff8b-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:55:42.341804 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:42.341756 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnkhj\" (UniqueName: \"kubernetes.io/projected/dd444e9c-1e51-4137-9768-00e4ad65ff8b-kube-api-access-pnkhj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:55:42.774767 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:42.774718 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" podUID="41ee9e23-5087-46d4-92d6-52b1e454a88b" containerName="payload-pre-processing" containerID="cri-o://0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a" gracePeriod=25 Nov 02 23:55:43.028875 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.028840 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:55:43.043933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.043906 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" Nov 02 23:55:43.044111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.043941 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq" event={"ID":"dd444e9c-1e51-4137-9768-00e4ad65ff8b","Type":"ContainerDied","Data":"4b4c4b11a04b0226c54800f3aa5379fe0d1d1e34ab0152d4614b8923e65345b7"} Nov 02 23:55:43.044111 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.043988 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b4c4b11a04b0226c54800f3aa5379fe0d1d1e34ab0152d4614b8923e65345b7" Nov 02 23:55:43.045504 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.045474 5672 generic.go:356] "Generic (PLEG): container finished" podID="41ee9e23-5087-46d4-92d6-52b1e454a88b" containerID="0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a" exitCode=0 Nov 02 23:55:43.045619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.045517 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" event={"ID":"41ee9e23-5087-46d4-92d6-52b1e454a88b","Type":"ContainerDied","Data":"0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a"} Nov 02 23:55:43.045619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.045555 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" event={"ID":"41ee9e23-5087-46d4-92d6-52b1e454a88b","Type":"ContainerDied","Data":"792d562da04965c88ef8edbf6d87695d1460b73f4553d743b11b2034d177cc8f"} Nov 02 23:55:43.045619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.045583 5672 scope.go:122] "RemoveContainer" containerID="0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a" Nov 02 23:55:43.045619 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.045519 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r" Nov 02 23:55:43.063506 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.063484 5672 scope.go:122] "RemoveContainer" containerID="0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a" Nov 02 23:55:43.063796 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:55:43.063770 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a\": container with ID starting with 0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a not found: ID does not exist" containerID="0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a" Nov 02 23:55:43.063868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.063798 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a"} err="failed to get container status \"0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a\": rpc error: code = NotFound desc = could not find container \"0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a\": container with ID starting with 0ef40aa303e5c810f213c6a76bbee49c4074331c1c60e713d66a290d5d46407a not found: ID does not exist" Nov 02 23:55:43.151757 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.151719 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ee9e23-5087-46d4-92d6-52b1e454a88b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41ee9e23-5087-46d4-92d6-52b1e454a88b-plugins-config-volume\") pod \"41ee9e23-5087-46d4-92d6-52b1e454a88b\" (UID: \"41ee9e23-5087-46d4-92d6-52b1e454a88b\") " Nov 02 23:55:43.151959 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.151843 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ee9e23-5087-46d4-92d6-52b1e454a88b-kube-api-access-dfbgv\" (UniqueName: \"kubernetes.io/projected/41ee9e23-5087-46d4-92d6-52b1e454a88b-kube-api-access-dfbgv\") pod \"41ee9e23-5087-46d4-92d6-52b1e454a88b\" (UID: \"41ee9e23-5087-46d4-92d6-52b1e454a88b\") " Nov 02 23:55:43.152435 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.152386 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ee9e23-5087-46d4-92d6-52b1e454a88b-plugins-config-volume" pod "41ee9e23-5087-46d4-92d6-52b1e454a88b" (UID: "41ee9e23-5087-46d4-92d6-52b1e454a88b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:55:43.154969 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.154935 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ee9e23-5087-46d4-92d6-52b1e454a88b-kube-api-access-dfbgv" pod "41ee9e23-5087-46d4-92d6-52b1e454a88b" (UID: "41ee9e23-5087-46d4-92d6-52b1e454a88b"). InnerVolumeSpecName "kube-api-access-dfbgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:55:43.253824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.253789 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfbgv\" (UniqueName: \"kubernetes.io/projected/41ee9e23-5087-46d4-92d6-52b1e454a88b-kube-api-access-dfbgv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:55:43.253824 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.253816 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41ee9e23-5087-46d4-92d6-52b1e454a88b-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:55:43.459285 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.459250 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r"] Nov 02 23:55:43.463173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:43.463131 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c2bff-85f97cff55-5cl4r"] Nov 02 23:55:44.712534 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:44.712502 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ee9e23-5087-46d4-92d6-52b1e454a88b" path="/var/lib/kubelet/pods/41ee9e23-5087-46d4-92d6-52b1e454a88b/volumes" Nov 02 23:55:47.047384 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:47.047354 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz"] Nov 02 23:55:47.093588 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:47.093553 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce38609-7sgtz"] Nov 02 23:55:48.713225 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:55:48.713191 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84aa4b8b-e6df-4e19-8fd9-c352394b5fcb" path="/var/lib/kubelet/pods/84aa4b8b-e6df-4e19-8fd9-c352394b5fcb/volumes" Nov 02 23:55:56.020618 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:55:56.020581 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:56:14.389491 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.389447 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg"] Nov 02 23:56:14.390243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.390220 5672 cpu_manager.go:414] "containerMap: removing container" podUID="dd444e9c-1e51-4137-9768-00e4ad65ff8b" containerName="revoke-keys" Nov 02 23:56:14.390243 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.390243 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd444e9c-1e51-4137-9768-00e4ad65ff8b" containerName="revoke-keys" Nov 02 23:56:14.390422 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.390315 5672 cpu_manager.go:414] "containerMap: removing container" podUID="41ee9e23-5087-46d4-92d6-52b1e454a88b" containerName="payload-pre-processing" Nov 02 23:56:14.390422 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.390325 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ee9e23-5087-46d4-92d6-52b1e454a88b" containerName="payload-pre-processing" Nov 02 23:56:14.390537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.390500 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ee9e23-5087-46d4-92d6-52b1e454a88b" containerName="payload-pre-processing" Nov 02 23:56:14.390537 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.390520 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd444e9c-1e51-4137-9768-00e4ad65ff8b" containerName="revoke-keys" Nov 02 23:56:14.394110 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.394087 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:14.447499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.447467 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg"] Nov 02 23:56:14.490249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.490216 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:56:14.490249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.490226 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:56:14.490463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.490253 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:56:14.490463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.490282 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:56:14.509361 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.509337 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c11b50-n85gg\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:14.509489 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.509379 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c11b50-n85gg\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:14.509489 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.509446 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drtlr\" (UniqueName: \"kubernetes.io/projected/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-kube-api-access-drtlr\") pod \"maas-api-revoke-keys-e2e-autores-c11b50-n85gg\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:14.610646 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.610616 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c11b50-n85gg\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:14.610792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.610658 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c11b50-n85gg\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:14.610792 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.610690 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drtlr\" (UniqueName: \"kubernetes.io/projected/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-kube-api-access-drtlr\") pod \"maas-api-revoke-keys-e2e-autores-c11b50-n85gg\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:14.611353 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.611325 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c11b50-n85gg\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:14.612971 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.612949 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c11b50-n85gg\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:14.663654 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.663589 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drtlr\" (UniqueName: \"kubernetes.io/projected/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-kube-api-access-drtlr\") pod \"maas-api-revoke-keys-e2e-autores-c11b50-n85gg\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:14.707881 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:14.707849 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:15.076365 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:15.076335 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg"] Nov 02 23:56:15.077885 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:56:15.077854 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8f967d_68f0_4f4c_8eb1_b158d8fbcb94.slice/crio-8e7a69c8f79301943da87e1e667f8bd9f3c178dd26039eff2ad2a7e731fd5249 WatchSource:0}: Error finding container 8e7a69c8f79301943da87e1e667f8bd9f3c178dd26039eff2ad2a7e731fd5249: Status 404 returned error can't find the container with id 8e7a69c8f79301943da87e1e667f8bd9f3c178dd26039eff2ad2a7e731fd5249 Nov 02 23:56:15.268555 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:15.268519 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" event={"ID":"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94","Type":"ContainerStarted","Data":"8e7a69c8f79301943da87e1e667f8bd9f3c178dd26039eff2ad2a7e731fd5249"} Nov 02 23:56:16.278500 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:16.278469 5672 generic.go:356] "Generic (PLEG): container finished" podID="ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94" containerID="68fa82454ba9165552a652912598512647ebc4c4794541312f5d6ba88ba5856c" exitCode=0 Nov 02 23:56:16.278892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:16.278556 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" event={"ID":"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94","Type":"ContainerDied","Data":"68fa82454ba9165552a652912598512647ebc4c4794541312f5d6ba88ba5856c"} Nov 02 23:56:17.450946 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:17.450918 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:17.543084 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:17.543013 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-kube-api-access-drtlr\" (UniqueName: \"kubernetes.io/projected/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-kube-api-access-drtlr\") pod \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " Nov 02 23:56:17.543249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:17.543130 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-xks-ca\") pod \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " Nov 02 23:56:17.543299 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:17.543276 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-openshift-ca\") pod \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\" (UID: \"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94\") " Nov 02 23:56:17.543724 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:17.543690 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-openshift-ca" pod "ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94" (UID: "ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:56:17.545276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:17.545246 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-xks-ca" pod "ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94" (UID: "ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:56:17.545386 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:17.545272 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-kube-api-access-drtlr" pod "ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94" (UID: "ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94"). InnerVolumeSpecName "kube-api-access-drtlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:56:17.644396 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:17.644364 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:17.644396 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:17.644391 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:17.644396 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:17.644400 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drtlr\" (UniqueName: \"kubernetes.io/projected/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94-kube-api-access-drtlr\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:18.293982 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:18.293951 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" Nov 02 23:56:18.293982 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:18.293970 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg" event={"ID":"ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94","Type":"ContainerDied","Data":"8e7a69c8f79301943da87e1e667f8bd9f3c178dd26039eff2ad2a7e731fd5249"} Nov 02 23:56:18.294290 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:18.294054 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e7a69c8f79301943da87e1e667f8bd9f3c178dd26039eff2ad2a7e731fd5249" Nov 02 23:56:26.021412 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:56:26.021362 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:56:29.172936 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:29.172906 5672 scope.go:122] "RemoveContainer" containerID="dbc95b6241da3dbc0b93e3400514e5415df10e4b3ff2206e047fe781df197e99" Nov 02 23:56:29.182598 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:29.182576 5672 scope.go:122] "RemoveContainer" containerID="e8d3c1ac22439762468a4e60b349a40b653e145a39fbfc3b7f178b642779f97f" Nov 02 23:56:29.191228 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:29.191203 5672 scope.go:122] "RemoveContainer" containerID="009c4231d978c87c9146f2472b2bedd63c368ca5fca9620954fb366dcba896e1" Nov 02 23:56:49.545808 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:49.545768 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z"] Nov 02 23:56:53.382826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:53.382792 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:56:54.546536 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:54.546478 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" podUID="ff85da83-99f1-472c-9c28-2e2f7a1f15ff" containerName="payload-pre-processing" containerID="cri-o://42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77" gracePeriod=25 Nov 02 23:56:54.815453 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:54.815391 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:56:54.965569 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:54.965538 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-plugins-config-volume\") pod \"ff85da83-99f1-472c-9c28-2e2f7a1f15ff\" (UID: \"ff85da83-99f1-472c-9c28-2e2f7a1f15ff\") " Nov 02 23:56:54.965776 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:54.965665 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-kube-api-access-cbgzt\" (UniqueName: \"kubernetes.io/projected/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-kube-api-access-cbgzt\") pod \"ff85da83-99f1-472c-9c28-2e2f7a1f15ff\" (UID: \"ff85da83-99f1-472c-9c28-2e2f7a1f15ff\") " Nov 02 23:56:54.965947 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:54.965924 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-plugins-config-volume" pod "ff85da83-99f1-472c-9c28-2e2f7a1f15ff" (UID: "ff85da83-99f1-472c-9c28-2e2f7a1f15ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:56:54.966109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:54.966094 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:54.967720 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:54.967694 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-kube-api-access-cbgzt" pod "ff85da83-99f1-472c-9c28-2e2f7a1f15ff" (UID: "ff85da83-99f1-472c-9c28-2e2f7a1f15ff"). InnerVolumeSpecName "kube-api-access-cbgzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:56:55.066989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:55.066912 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbgzt\" (UniqueName: \"kubernetes.io/projected/ff85da83-99f1-472c-9c28-2e2f7a1f15ff-kube-api-access-cbgzt\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:56:55.543559 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:55.543525 5672 generic.go:356] "Generic (PLEG): container finished" podID="ff85da83-99f1-472c-9c28-2e2f7a1f15ff" containerID="42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77" exitCode=0 Nov 02 23:56:55.543752 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:55.543578 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" Nov 02 23:56:55.543752 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:55.543590 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" event={"ID":"ff85da83-99f1-472c-9c28-2e2f7a1f15ff","Type":"ContainerDied","Data":"42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77"} Nov 02 23:56:55.543752 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:55.543634 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z" event={"ID":"ff85da83-99f1-472c-9c28-2e2f7a1f15ff","Type":"ContainerDied","Data":"6018a2fdeaa89a619fd17d031e55212056c57ddc7f803a7389e78660617623f0"} Nov 02 23:56:55.543752 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:55.543654 5672 scope.go:122] "RemoveContainer" containerID="42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77" Nov 02 23:56:55.554481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:55.554321 5672 scope.go:122] "RemoveContainer" containerID="42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77" Nov 02 23:56:55.554723 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:56:55.554584 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77\": container with ID starting with 42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77 not found: ID does not exist" containerID="42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77" Nov 02 23:56:55.554723 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:55.554617 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77"} err="failed to get container status \"42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77\": rpc error: code = NotFound desc = could not find container \"42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77\": container with ID starting with 42a7f64cb7a41b30e5d583640fa653710623dbf8c12566aa934404908711fe77 not found: ID does not exist" Nov 02 23:56:55.696298 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:55.696266 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z"] Nov 02 23:56:55.742582 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:55.742553 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-661a20-89c6cf9b7-fmz6z"] Nov 02 23:56:56.058995 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:56:56.058923 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:56:56.714672 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:56:56.714640 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff85da83-99f1-472c-9c28-2e2f7a1f15ff" path="/var/lib/kubelet/pods/ff85da83-99f1-472c-9c28-2e2f7a1f15ff/volumes" Nov 02 23:57:06.554038 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:06.553998 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm"] Nov 02 23:57:06.653615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:06.653578 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d"] Nov 02 23:57:09.333672 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:09.333633 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq"] Nov 02 23:57:09.378366 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:09.378332 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f2df33-jmlnq"] Nov 02 23:57:09.557871 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:09.557828 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd"] Nov 02 23:57:09.558514 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:09.558456 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" containerID="cri-o://e4c0d55570e03210ca0add0e9e6709853b6d7c1307e605050b6a6e255dd360b4" gracePeriod=30 Nov 02 23:57:10.713253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:10.713213 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd444e9c-1e51-4137-9768-00e4ad65ff8b" path="/var/lib/kubelet/pods/dd444e9c-1e51-4137-9768-00e4ad65ff8b/volumes" Nov 02 23:57:11.555085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:11.555038 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" podUID="e7c2e004-7a8a-431d-8380-b96ba2f0a628" containerName="payload-processing" containerID="cri-o://ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79" gracePeriod=25 Nov 02 23:57:11.654737 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:11.654688 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" podUID="ec024bb7-0d87-4472-9cd9-4be146118f14" containerName="payload-pre-processing" containerID="cri-o://52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd" gracePeriod=25 Nov 02 23:57:11.811285 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:11.811220 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:57:11.901892 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:11.901854 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c2e004-7a8a-431d-8380-b96ba2f0a628-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c2e004-7a8a-431d-8380-b96ba2f0a628-plugins-config-volume\") pod \"e7c2e004-7a8a-431d-8380-b96ba2f0a628\" (UID: \"e7c2e004-7a8a-431d-8380-b96ba2f0a628\") " Nov 02 23:57:11.902082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:11.902017 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c2e004-7a8a-431d-8380-b96ba2f0a628-kube-api-access-pxn99\" (UniqueName: \"kubernetes.io/projected/e7c2e004-7a8a-431d-8380-b96ba2f0a628-kube-api-access-pxn99\") pod \"e7c2e004-7a8a-431d-8380-b96ba2f0a628\" (UID: \"e7c2e004-7a8a-431d-8380-b96ba2f0a628\") " Nov 02 23:57:11.902355 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:11.902317 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c2e004-7a8a-431d-8380-b96ba2f0a628-plugins-config-volume" pod "e7c2e004-7a8a-431d-8380-b96ba2f0a628" (UID: "e7c2e004-7a8a-431d-8380-b96ba2f0a628"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:11.902481 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:11.902465 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c2e004-7a8a-431d-8380-b96ba2f0a628-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:11.904173 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:11.904135 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c2e004-7a8a-431d-8380-b96ba2f0a628-kube-api-access-pxn99" pod "e7c2e004-7a8a-431d-8380-b96ba2f0a628" (UID: "e7c2e004-7a8a-431d-8380-b96ba2f0a628"). InnerVolumeSpecName "kube-api-access-pxn99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:11.926039 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:11.926011 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:57:12.003133 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.003100 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec024bb7-0d87-4472-9cd9-4be146118f14-kube-api-access-48fbp\" (UniqueName: \"kubernetes.io/projected/ec024bb7-0d87-4472-9cd9-4be146118f14-kube-api-access-48fbp\") pod \"ec024bb7-0d87-4472-9cd9-4be146118f14\" (UID: \"ec024bb7-0d87-4472-9cd9-4be146118f14\") " Nov 02 23:57:12.003320 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.003279 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec024bb7-0d87-4472-9cd9-4be146118f14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec024bb7-0d87-4472-9cd9-4be146118f14-plugins-config-volume\") pod \"ec024bb7-0d87-4472-9cd9-4be146118f14\" (UID: \"ec024bb7-0d87-4472-9cd9-4be146118f14\") " Nov 02 23:57:12.003567 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.003552 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxn99\" (UniqueName: \"kubernetes.io/projected/e7c2e004-7a8a-431d-8380-b96ba2f0a628-kube-api-access-pxn99\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:12.003705 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.003682 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec024bb7-0d87-4472-9cd9-4be146118f14-plugins-config-volume" pod "ec024bb7-0d87-4472-9cd9-4be146118f14" (UID: "ec024bb7-0d87-4472-9cd9-4be146118f14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:12.005333 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.005264 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec024bb7-0d87-4472-9cd9-4be146118f14-kube-api-access-48fbp" pod "ec024bb7-0d87-4472-9cd9-4be146118f14" (UID: "ec024bb7-0d87-4472-9cd9-4be146118f14"). InnerVolumeSpecName "kube-api-access-48fbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:12.104873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.104799 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec024bb7-0d87-4472-9cd9-4be146118f14-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:12.104873 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.104826 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48fbp\" (UniqueName: \"kubernetes.io/projected/ec024bb7-0d87-4472-9cd9-4be146118f14-kube-api-access-48fbp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:12.674578 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.674538 5672 generic.go:356] "Generic (PLEG): container finished" podID="ec024bb7-0d87-4472-9cd9-4be146118f14" containerID="52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd" exitCode=0 Nov 02 23:57:12.674777 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.674599 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" Nov 02 23:57:12.674777 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.674622 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" event={"ID":"ec024bb7-0d87-4472-9cd9-4be146118f14","Type":"ContainerDied","Data":"52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd"} Nov 02 23:57:12.674777 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.674683 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d" event={"ID":"ec024bb7-0d87-4472-9cd9-4be146118f14","Type":"ContainerDied","Data":"e021505c4b27ad89d8d9b3a4878356cec6055639a53858dc156f54e27e607981"} Nov 02 23:57:12.674777 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.674712 5672 scope.go:122] "RemoveContainer" containerID="52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd" Nov 02 23:57:12.676383 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.676360 5672 generic.go:356] "Generic (PLEG): container finished" podID="e7c2e004-7a8a-431d-8380-b96ba2f0a628" containerID="ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79" exitCode=0 Nov 02 23:57:12.676527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.676408 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" Nov 02 23:57:12.676527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.676446 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" event={"ID":"e7c2e004-7a8a-431d-8380-b96ba2f0a628","Type":"ContainerDied","Data":"ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79"} Nov 02 23:57:12.676527 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.676500 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm" event={"ID":"e7c2e004-7a8a-431d-8380-b96ba2f0a628","Type":"ContainerDied","Data":"4417ad342dc5c620ea414cc89a8ab4e88e37693aa420d7933437b9370c3ed5ee"} Nov 02 23:57:12.688814 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.688791 5672 scope.go:122] "RemoveContainer" containerID="52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd" Nov 02 23:57:12.689219 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:57:12.689192 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd\": container with ID starting with 52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd not found: ID does not exist" containerID="52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd" Nov 02 23:57:12.689363 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.689335 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd"} err="failed to get container status \"52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd\": rpc error: code = NotFound desc = could not find container \"52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd\": container with ID starting with 52134c0b56b8b4d990dace2b1eb1c48d45129ff7ff5c84ce5c117bae7df47bfd not found: ID does not exist" Nov 02 23:57:12.689462 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.689365 5672 scope.go:122] "RemoveContainer" containerID="ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79" Nov 02 23:57:12.701897 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.701875 5672 scope.go:122] "RemoveContainer" containerID="ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79" Nov 02 23:57:12.702234 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:57:12.702204 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79\": container with ID starting with ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79 not found: ID does not exist" containerID="ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79" Nov 02 23:57:12.702336 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.702242 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79"} err="failed to get container status \"ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79\": rpc error: code = NotFound desc = could not find container \"ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79\": container with ID starting with ae7528274fc2a61d64fdc1174d1dea880df3b20bce58b182094c562456e1af79 not found: ID does not exist" Nov 02 23:57:12.829697 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.829572 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d"] Nov 02 23:57:12.877584 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.877544 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4b63d8-549cb98699-jhk5d"] Nov 02 23:57:12.977930 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.977892 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm"] Nov 02 23:57:12.982004 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:12.981974 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4b63d8-59566ffdc5-fnxqm"] Nov 02 23:57:14.698038 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.697998 5672 generic.go:356] "Generic (PLEG): container finished" podID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerID="e4c0d55570e03210ca0add0e9e6709853b6d7c1307e605050b6a6e255dd360b4" exitCode=0 Nov 02 23:57:14.698482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.698070 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" event={"ID":"e5dd7cdf-8f73-48dc-a25e-507296833d93","Type":"ContainerDied","Data":"e4c0d55570e03210ca0add0e9e6709853b6d7c1307e605050b6a6e255dd360b4"} Nov 02 23:57:14.698482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.698159 5672 scope.go:122] "RemoveContainer" containerID="900c6b114b6e4d1976350fe0429c3052e5488714502315978c3c297af32ba5e5" Nov 02 23:57:14.713856 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.713829 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c2e004-7a8a-431d-8380-b96ba2f0a628" path="/var/lib/kubelet/pods/e7c2e004-7a8a-431d-8380-b96ba2f0a628/volumes" Nov 02 23:57:14.714242 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.714227 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec024bb7-0d87-4472-9cd9-4be146118f14" path="/var/lib/kubelet/pods/ec024bb7-0d87-4472-9cd9-4be146118f14/volumes" Nov 02 23:57:14.833958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.833899 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:57:14.950725 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.950690 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-socket\") pod \"e5dd7cdf-8f73-48dc-a25e-507296833d93\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " Nov 02 23:57:14.950925 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.950741 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-envoy\") pod \"e5dd7cdf-8f73-48dc-a25e-507296833d93\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " Nov 02 23:57:14.950925 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.950764 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5dd7cdf-8f73-48dc-a25e-507296833d93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5dd7cdf-8f73-48dc-a25e-507296833d93-istiod-ca-cert\") pod \"e5dd7cdf-8f73-48dc-a25e-507296833d93\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " Nov 02 23:57:14.950925 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.950790 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-podinfo\") pod \"e5dd7cdf-8f73-48dc-a25e-507296833d93\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " Nov 02 23:57:14.950925 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.950815 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-data\") pod \"e5dd7cdf-8f73-48dc-a25e-507296833d93\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " Nov 02 23:57:14.950925 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.950896 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-certs\") pod \"e5dd7cdf-8f73-48dc-a25e-507296833d93\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " Nov 02 23:57:14.951289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.950972 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-credential-socket\") pod \"e5dd7cdf-8f73-48dc-a25e-507296833d93\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " Nov 02 23:57:14.951289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.951008 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-socket" pod "e5dd7cdf-8f73-48dc-a25e-507296833d93" (UID: "e5dd7cdf-8f73-48dc-a25e-507296833d93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:14.951289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.951085 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-token\" (UniqueName: \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-token\") pod \"e5dd7cdf-8f73-48dc-a25e-507296833d93\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " Nov 02 23:57:14.951289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.951120 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-kube-api-access-wj2rv\" (UniqueName: \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-kube-api-access-wj2rv\") pod \"e5dd7cdf-8f73-48dc-a25e-507296833d93\" (UID: \"e5dd7cdf-8f73-48dc-a25e-507296833d93\") " Nov 02 23:57:14.951289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.951240 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-certs" pod "e5dd7cdf-8f73-48dc-a25e-507296833d93" (UID: "e5dd7cdf-8f73-48dc-a25e-507296833d93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:14.951289 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.951277 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5dd7cdf-8f73-48dc-a25e-507296833d93-istiod-ca-cert" pod "e5dd7cdf-8f73-48dc-a25e-507296833d93" (UID: "e5dd7cdf-8f73-48dc-a25e-507296833d93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:14.951665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.951366 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-credential-socket" pod "e5dd7cdf-8f73-48dc-a25e-507296833d93" (UID: "e5dd7cdf-8f73-48dc-a25e-507296833d93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:14.951727 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.951661 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:14.951727 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.951683 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5dd7cdf-8f73-48dc-a25e-507296833d93-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:14.951727 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.951696 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:14.951727 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.951713 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:14.951949 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.951781 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-data" pod "e5dd7cdf-8f73-48dc-a25e-507296833d93" (UID: "e5dd7cdf-8f73-48dc-a25e-507296833d93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:14.953138 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.953104 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-podinfo" pod "e5dd7cdf-8f73-48dc-a25e-507296833d93" (UID: "e5dd7cdf-8f73-48dc-a25e-507296833d93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:57:14.953407 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.953385 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-envoy" pod "e5dd7cdf-8f73-48dc-a25e-507296833d93" (UID: "e5dd7cdf-8f73-48dc-a25e-507296833d93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:14.953528 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.953509 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-kube-api-access-wj2rv" pod "e5dd7cdf-8f73-48dc-a25e-507296833d93" (UID: "e5dd7cdf-8f73-48dc-a25e-507296833d93"). InnerVolumeSpecName "kube-api-access-wj2rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:14.953895 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:14.953870 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-token" pod "e5dd7cdf-8f73-48dc-a25e-507296833d93" (UID: "e5dd7cdf-8f73-48dc-a25e-507296833d93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:15.052654 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:15.052623 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:15.052654 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:15.052651 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:15.052654 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:15.052660 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:15.052970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:15.052670 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj2rv\" (UniqueName: \"kubernetes.io/projected/e5dd7cdf-8f73-48dc-a25e-507296833d93-kube-api-access-wj2rv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:15.052970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:15.052681 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5dd7cdf-8f73-48dc-a25e-507296833d93-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:15.706080 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:15.706047 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" Nov 02 23:57:15.706569 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:15.706043 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd" event={"ID":"e5dd7cdf-8f73-48dc-a25e-507296833d93","Type":"ContainerDied","Data":"76d71118df91742576b3a07e02447c337714eb07c868255a968d3cb7ba02c488"} Nov 02 23:57:15.706569 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:15.706195 5672 scope.go:122] "RemoveContainer" containerID="e4c0d55570e03210ca0add0e9e6709853b6d7c1307e605050b6a6e255dd360b4" Nov 02 23:57:15.906376 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:15.906339 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd"] Nov 02 23:57:15.952719 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:15.952686 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f2df33-openshift-default-646b777445-w5rkd"] Nov 02 23:57:16.446814 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.446776 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh"] Nov 02 23:57:16.447343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447327 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" Nov 02 23:57:16.447392 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447343 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" Nov 02 23:57:16.447392 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447357 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ec024bb7-0d87-4472-9cd9-4be146118f14" containerName="payload-pre-processing" Nov 02 23:57:16.447392 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447362 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec024bb7-0d87-4472-9cd9-4be146118f14" containerName="payload-pre-processing" Nov 02 23:57:16.447392 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447378 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ff85da83-99f1-472c-9c28-2e2f7a1f15ff" containerName="payload-pre-processing" Nov 02 23:57:16.447392 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447383 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff85da83-99f1-472c-9c28-2e2f7a1f15ff" containerName="payload-pre-processing" Nov 02 23:57:16.447589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447398 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94" containerName="revoke-keys" Nov 02 23:57:16.447589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447403 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94" containerName="revoke-keys" Nov 02 23:57:16.447589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447413 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" Nov 02 23:57:16.447589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447417 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" Nov 02 23:57:16.447589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447433 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e7c2e004-7a8a-431d-8380-b96ba2f0a628" containerName="payload-processing" Nov 02 23:57:16.447589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447437 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c2e004-7a8a-431d-8380-b96ba2f0a628" containerName="payload-processing" Nov 02 23:57:16.447589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447509 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94" containerName="revoke-keys" Nov 02 23:57:16.447589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447516 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" Nov 02 23:57:16.447589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447524 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c2e004-7a8a-431d-8380-b96ba2f0a628" containerName="payload-processing" Nov 02 23:57:16.447589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447534 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec024bb7-0d87-4472-9cd9-4be146118f14" containerName="payload-pre-processing" Nov 02 23:57:16.447589 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.447542 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff85da83-99f1-472c-9c28-2e2f7a1f15ff" containerName="payload-pre-processing" Nov 02 23:57:16.452302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.452280 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.501123 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.501093 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-eb3055757321\"" Nov 02 23:57:16.501502 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.501478 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a610cb5fdd81\"" Nov 02 23:57:16.501628 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.501556 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 23:57:16.501628 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.501574 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-nqg67\"" Nov 02 23:57:16.501628 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.501610 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 23:57:16.501782 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.501640 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 23:57:16.501782 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.501665 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 23:57:16.504825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.504803 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh"] Nov 02 23:57:16.570573 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.570541 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2wvd\" (UniqueName: \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-kube-api-access-c2wvd\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.570769 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.570584 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-plugins-config-volume\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.570769 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.570613 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-routing-overlay-volume\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.570907 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.570780 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-provider-credentials\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.671840 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.671806 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wvd\" (UniqueName: \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-kube-api-access-c2wvd\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.672062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.671860 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-plugins-config-volume\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.672062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.671916 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-routing-overlay-volume\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.672062 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.671975 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-provider-credentials\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.672641 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.672618 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-routing-overlay-volume\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.672826 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.672802 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-plugins-config-volume\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.674521 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.674499 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-provider-credentials\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.712496 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.712419 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" path="/var/lib/kubelet/pods/e5dd7cdf-8f73-48dc-a25e-507296833d93/volumes" Nov 02 23:57:16.725847 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.725815 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2wvd\" (UniqueName: \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-kube-api-access-c2wvd\") pod \"payload-processing-external-model-6cf9bbf985-pvbkh\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:16.765337 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:16.765307 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:17.141172 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:17.141105 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh"] Nov 02 23:57:17.142308 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:57:17.142259 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e81664_a4d8_4991_9984_4beb393a31aa.slice/crio-898bfc21976fe49e0f58a49ffbd84b4c086269d078160cec445e37af87806782 WatchSource:0}: Error finding container 898bfc21976fe49e0f58a49ffbd84b4c086269d078160cec445e37af87806782: Status 404 returned error can't find the container with id 898bfc21976fe49e0f58a49ffbd84b4c086269d078160cec445e37af87806782 Nov 02 23:57:17.722195 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:17.722130 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" event={"ID":"d9e81664-a4d8-4991-9984-4beb393a31aa","Type":"ContainerStarted","Data":"418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65"} Nov 02 23:57:17.722598 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:17.722217 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" event={"ID":"d9e81664-a4d8-4991-9984-4beb393a31aa","Type":"ContainerStarted","Data":"898bfc21976fe49e0f58a49ffbd84b4c086269d078160cec445e37af87806782"} Nov 02 23:57:17.722598 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:17.722246 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:17.829300 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:17.829249 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" podStartSLOduration=1.829234543 podStartE2EDuration="1.829234543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:57:17.828276191 +0000 UTC m=+3831.235588021" watchObservedRunningTime="2026-11-02 23:57:17.829234543 +0000 UTC m=+3831.236546373" Nov 02 23:57:21.300179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:21.300128 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh"] Nov 02 23:57:26.020505 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:57:26.020457 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:57:26.300522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.300434 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" podUID="d9e81664-a4d8-4991-9984-4beb393a31aa" containerName="payload-processing" containerID="cri-o://418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65" gracePeriod=25 Nov 02 23:57:26.580478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.580451 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:26.689078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.689036 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-provider-credentials\") pod \"d9e81664-a4d8-4991-9984-4beb393a31aa\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " Nov 02 23:57:26.689280 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.689209 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-routing-overlay-volume\") pod \"d9e81664-a4d8-4991-9984-4beb393a31aa\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " Nov 02 23:57:26.689341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.689285 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-plugins-config-volume\") pod \"d9e81664-a4d8-4991-9984-4beb393a31aa\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " Nov 02 23:57:26.689341 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.689327 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-kube-api-access-c2wvd\" (UniqueName: \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-kube-api-access-c2wvd\") pod \"d9e81664-a4d8-4991-9984-4beb393a31aa\" (UID: \"d9e81664-a4d8-4991-9984-4beb393a31aa\") " Nov 02 23:57:26.689637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.689592 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-routing-overlay-volume" pod "d9e81664-a4d8-4991-9984-4beb393a31aa" (UID: "d9e81664-a4d8-4991-9984-4beb393a31aa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:26.689831 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.689814 5672 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-routing-overlay-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:26.689896 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.689821 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-plugins-config-volume" pod "d9e81664-a4d8-4991-9984-4beb393a31aa" (UID: "d9e81664-a4d8-4991-9984-4beb393a31aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:26.691422 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.691402 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-provider-credentials" pod "d9e81664-a4d8-4991-9984-4beb393a31aa" (UID: "d9e81664-a4d8-4991-9984-4beb393a31aa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:26.691887 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.691862 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-kube-api-access-c2wvd" pod "d9e81664-a4d8-4991-9984-4beb393a31aa" (UID: "d9e81664-a4d8-4991-9984-4beb393a31aa"). InnerVolumeSpecName "kube-api-access-c2wvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:26.786499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.786463 5672 generic.go:356] "Generic (PLEG): container finished" podID="d9e81664-a4d8-4991-9984-4beb393a31aa" containerID="418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65" exitCode=0 Nov 02 23:57:26.786665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.786515 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" Nov 02 23:57:26.786665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.786545 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" event={"ID":"d9e81664-a4d8-4991-9984-4beb393a31aa","Type":"ContainerDied","Data":"418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65"} Nov 02 23:57:26.786665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.786600 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh" event={"ID":"d9e81664-a4d8-4991-9984-4beb393a31aa","Type":"ContainerDied","Data":"898bfc21976fe49e0f58a49ffbd84b4c086269d078160cec445e37af87806782"} Nov 02 23:57:26.786665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.786621 5672 scope.go:122] "RemoveContainer" containerID="418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65" Nov 02 23:57:26.790848 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.790828 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e81664-a4d8-4991-9984-4beb393a31aa-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:26.790997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.790851 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2wvd\" (UniqueName: \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-kube-api-access-c2wvd\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:26.790997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.790866 5672 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9e81664-a4d8-4991-9984-4beb393a31aa-provider-credentials\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:26.798707 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.798680 5672 scope.go:122] "RemoveContainer" containerID="418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65" Nov 02 23:57:26.798990 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:57:26.798970 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65\": container with ID starting with 418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65 not found: ID does not exist" containerID="418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65" Nov 02 23:57:26.799050 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.799000 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65"} err="failed to get container status \"418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65\": rpc error: code = NotFound desc = could not find container \"418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65\": container with ID starting with 418d4fa6bdfbfc633c68045ad2b253c14f0f9859cffdc8c54fb0890599a78b65 not found: ID does not exist" Nov 02 23:57:26.944904 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.944817 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh"] Nov 02 23:57:26.948249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:26.948211 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cf9bbf985-pvbkh"] Nov 02 23:57:28.713525 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:28.713482 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e81664-a4d8-4991-9984-4beb393a31aa" path="/var/lib/kubelet/pods/d9e81664-a4d8-4991-9984-4beb393a31aa/volumes" Nov 02 23:57:29.229639 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:29.229595 5672 scope.go:122] "RemoveContainer" containerID="1ddf9df3ee7a7328bfb0447d01985528148d98d9466096bee430051227a76cd9" Nov 02 23:57:29.239731 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:29.239707 5672 scope.go:122] "RemoveContainer" containerID="149f80064724c4c71b62057612ee556b9761c5477ecbc0b6458a98c2f0e4075c" Nov 02 23:57:43.114366 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:43.114322 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg"] Nov 02 23:57:43.158942 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:43.158883 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c11b50-n85gg"] Nov 02 23:57:44.234674 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.234640 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw"] Nov 02 23:57:44.235384 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.235361 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d9e81664-a4d8-4991-9984-4beb393a31aa" containerName="payload-processing" Nov 02 23:57:44.235498 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.235384 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e81664-a4d8-4991-9984-4beb393a31aa" containerName="payload-processing" Nov 02 23:57:44.235602 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.235587 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5dd7cdf-8f73-48dc-a25e-507296833d93" containerName="istio-proxy" Nov 02 23:57:44.235665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.235616 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e81664-a4d8-4991-9984-4beb393a31aa" containerName="payload-processing" Nov 02 23:57:44.239235 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.239211 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:44.292006 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.291971 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw"] Nov 02 23:57:44.323257 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.323220 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2dsl\" (UniqueName: \"kubernetes.io/projected/1663304d-daba-420d-8c7c-bd104b1ee141-kube-api-access-d2dsl\") pod \"maas-api-revoke-sub-2e01a905-sc5pw\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:44.323407 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.323369 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e01a905-sc5pw\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:44.323522 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.323502 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e01a905-sc5pw\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:44.335355 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.335327 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:57:44.335355 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.335348 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:57:44.335533 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.335338 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:57:44.335533 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.335331 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:57:44.424698 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.424656 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e01a905-sc5pw\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:44.424882 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.424757 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2dsl\" (UniqueName: \"kubernetes.io/projected/1663304d-daba-420d-8c7c-bd104b1ee141-kube-api-access-d2dsl\") pod \"maas-api-revoke-sub-2e01a905-sc5pw\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:44.424882 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.424830 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e01a905-sc5pw\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:44.425451 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.425422 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e01a905-sc5pw\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:44.427532 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.427505 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e01a905-sc5pw\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:44.478880 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.478852 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2dsl\" (UniqueName: \"kubernetes.io/projected/1663304d-daba-420d-8c7c-bd104b1ee141-kube-api-access-d2dsl\") pod \"maas-api-revoke-sub-2e01a905-sc5pw\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:44.555645 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.555568 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:44.713230 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.713203 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94" path="/var/lib/kubelet/pods/ac8f967d-68f0-4f4c-8eb1-b158d8fbcb94/volumes" Nov 02 23:57:44.921566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:44.921541 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw"] Nov 02 23:57:44.922231 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:57:44.922203 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1663304d_daba_420d_8c7c_bd104b1ee141.slice/crio-281ca02dfad79ee3d1eda20c1f1e225b1df79cbd1c76b6dd56060aa1a420c582 WatchSource:0}: Error finding container 281ca02dfad79ee3d1eda20c1f1e225b1df79cbd1c76b6dd56060aa1a420c582: Status 404 returned error can't find the container with id 281ca02dfad79ee3d1eda20c1f1e225b1df79cbd1c76b6dd56060aa1a420c582 Nov 02 23:57:45.922349 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:45.922304 5672 generic.go:356] "Generic (PLEG): container finished" podID="1663304d-daba-420d-8c7c-bd104b1ee141" containerID="4a4cde780674cf5dcd5e0a1490321c8e60812e1285f1f1bd2e2d960f676b8f46" exitCode=0 Nov 02 23:57:45.922773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:45.922353 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" event={"ID":"1663304d-daba-420d-8c7c-bd104b1ee141","Type":"ContainerDied","Data":"4a4cde780674cf5dcd5e0a1490321c8e60812e1285f1f1bd2e2d960f676b8f46"} Nov 02 23:57:45.922773 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:45.922407 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" event={"ID":"1663304d-daba-420d-8c7c-bd104b1ee141","Type":"ContainerStarted","Data":"281ca02dfad79ee3d1eda20c1f1e225b1df79cbd1c76b6dd56060aa1a420c582"} Nov 02 23:57:47.092842 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.092807 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:47.168841 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.168802 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-xks-ca\") pod \"1663304d-daba-420d-8c7c-bd104b1ee141\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " Nov 02 23:57:47.169275 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.169252 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1663304d-daba-420d-8c7c-bd104b1ee141-kube-api-access-d2dsl\" (UniqueName: \"kubernetes.io/projected/1663304d-daba-420d-8c7c-bd104b1ee141-kube-api-access-d2dsl\") pod \"1663304d-daba-420d-8c7c-bd104b1ee141\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " Nov 02 23:57:47.169463 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.169446 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-openshift-ca\") pod \"1663304d-daba-420d-8c7c-bd104b1ee141\" (UID: \"1663304d-daba-420d-8c7c-bd104b1ee141\") " Nov 02 23:57:47.169887 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.169843 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-openshift-ca" pod "1663304d-daba-420d-8c7c-bd104b1ee141" (UID: "1663304d-daba-420d-8c7c-bd104b1ee141"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:47.170303 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.170281 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:47.171639 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.171615 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-xks-ca" pod "1663304d-daba-420d-8c7c-bd104b1ee141" (UID: "1663304d-daba-420d-8c7c-bd104b1ee141"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:57:47.173544 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.173502 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1663304d-daba-420d-8c7c-bd104b1ee141-kube-api-access-d2dsl" pod "1663304d-daba-420d-8c7c-bd104b1ee141" (UID: "1663304d-daba-420d-8c7c-bd104b1ee141"). InnerVolumeSpecName "kube-api-access-d2dsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:47.273075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.273039 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1663304d-daba-420d-8c7c-bd104b1ee141-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:47.273075 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.273073 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2dsl\" (UniqueName: \"kubernetes.io/projected/1663304d-daba-420d-8c7c-bd104b1ee141-kube-api-access-d2dsl\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:47.939647 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.939599 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" event={"ID":"1663304d-daba-420d-8c7c-bd104b1ee141","Type":"ContainerDied","Data":"281ca02dfad79ee3d1eda20c1f1e225b1df79cbd1c76b6dd56060aa1a420c582"} Nov 02 23:57:47.939849 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.939666 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="281ca02dfad79ee3d1eda20c1f1e225b1df79cbd1c76b6dd56060aa1a420c582" Nov 02 23:57:47.939849 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:47.939708 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw" Nov 02 23:57:54.248623 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:54.248584 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t"] Nov 02 23:57:54.249064 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:54.248776 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" podUID="a3cff8c2-f122-4b13-a292-31bf15b6098b" containerName="istio-proxy" containerID="cri-o://ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754" gracePeriod=30 Nov 02 23:57:56.020411 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:57:56.020375 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:57:59.596737 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.596709 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:57:59.720278 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720239 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3cff8c2-f122-4b13-a292-31bf15b6098b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3cff8c2-f122-4b13-a292-31bf15b6098b-istiod-ca-cert\") pod \"a3cff8c2-f122-4b13-a292-31bf15b6098b\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " Nov 02 23:57:59.720478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720372 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-podinfo\") pod \"a3cff8c2-f122-4b13-a292-31bf15b6098b\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " Nov 02 23:57:59.720478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720401 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-data\") pod \"a3cff8c2-f122-4b13-a292-31bf15b6098b\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " Nov 02 23:57:59.720478 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720435 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-credential-socket\") pod \"a3cff8c2-f122-4b13-a292-31bf15b6098b\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " Nov 02 23:57:59.720662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720485 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-certs\") pod \"a3cff8c2-f122-4b13-a292-31bf15b6098b\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " Nov 02 23:57:59.720662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720518 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-socket\") pod \"a3cff8c2-f122-4b13-a292-31bf15b6098b\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " Nov 02 23:57:59.720662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720569 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-kube-api-access-bw8vc\" (UniqueName: \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-kube-api-access-bw8vc\") pod \"a3cff8c2-f122-4b13-a292-31bf15b6098b\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " Nov 02 23:57:59.720662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720601 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-token\" (UniqueName: \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-token\") pod \"a3cff8c2-f122-4b13-a292-31bf15b6098b\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " Nov 02 23:57:59.720662 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720641 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-envoy\") pod \"a3cff8c2-f122-4b13-a292-31bf15b6098b\" (UID: \"a3cff8c2-f122-4b13-a292-31bf15b6098b\") " Nov 02 23:57:59.720931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720597 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3cff8c2-f122-4b13-a292-31bf15b6098b-istiod-ca-cert" pod "a3cff8c2-f122-4b13-a292-31bf15b6098b" (UID: "a3cff8c2-f122-4b13-a292-31bf15b6098b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:57:59.720931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720730 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-data" pod "a3cff8c2-f122-4b13-a292-31bf15b6098b" (UID: "a3cff8c2-f122-4b13-a292-31bf15b6098b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:59.720931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720740 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-socket" pod "a3cff8c2-f122-4b13-a292-31bf15b6098b" (UID: "a3cff8c2-f122-4b13-a292-31bf15b6098b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:59.720931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720910 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-certs" pod "a3cff8c2-f122-4b13-a292-31bf15b6098b" (UID: "a3cff8c2-f122-4b13-a292-31bf15b6098b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:59.721168 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.720971 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-credential-socket" pod "a3cff8c2-f122-4b13-a292-31bf15b6098b" (UID: "a3cff8c2-f122-4b13-a292-31bf15b6098b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:59.721234 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.721210 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3cff8c2-f122-4b13-a292-31bf15b6098b-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:59.721234 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.721230 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:59.721344 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.721244 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:59.721344 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.721257 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:59.721344 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.721270 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:59.722934 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.722881 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-kube-api-access-bw8vc" pod "a3cff8c2-f122-4b13-a292-31bf15b6098b" (UID: "a3cff8c2-f122-4b13-a292-31bf15b6098b"). InnerVolumeSpecName "kube-api-access-bw8vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:59.723094 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.723050 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-podinfo" pod "a3cff8c2-f122-4b13-a292-31bf15b6098b" (UID: "a3cff8c2-f122-4b13-a292-31bf15b6098b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 23:57:59.723364 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.723337 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-envoy" pod "a3cff8c2-f122-4b13-a292-31bf15b6098b" (UID: "a3cff8c2-f122-4b13-a292-31bf15b6098b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 23:57:59.723480 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.723370 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-token" pod "a3cff8c2-f122-4b13-a292-31bf15b6098b" (UID: "a3cff8c2-f122-4b13-a292-31bf15b6098b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:57:59.823162 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.823067 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:59.823162 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.823102 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw8vc\" (UniqueName: \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-kube-api-access-bw8vc\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:59.823162 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.823117 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:57:59.823162 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:57:59.823131 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3cff8c2-f122-4b13-a292-31bf15b6098b-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:58:00.023637 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:00.023605 5672 generic.go:356] "Generic (PLEG): container finished" podID="a3cff8c2-f122-4b13-a292-31bf15b6098b" containerID="ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754" exitCode=0 Nov 02 23:58:00.023813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:00.023649 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" Nov 02 23:58:00.023813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:00.023689 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" event={"ID":"a3cff8c2-f122-4b13-a292-31bf15b6098b","Type":"ContainerDied","Data":"ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754"} Nov 02 23:58:00.023813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:00.023745 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t" event={"ID":"a3cff8c2-f122-4b13-a292-31bf15b6098b","Type":"ContainerDied","Data":"cec7cff5c6c2e1341874b42952b7d5f60b0eebc935864c6d643e87ae39040768"} Nov 02 23:58:00.023813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:00.023772 5672 scope.go:122] "RemoveContainer" containerID="ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754" Nov 02 23:58:00.037818 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:00.037797 5672 scope.go:122] "RemoveContainer" containerID="ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754" Nov 02 23:58:00.038128 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:58:00.038107 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754\": container with ID starting with ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754 not found: ID does not exist" containerID="ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754" Nov 02 23:58:00.038203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:00.038137 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754"} err="failed to get container status \"ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754\": rpc error: code = NotFound desc = could not find container \"ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754\": container with ID starting with ef856ccdc419ddee4918c6b6064635315b7efbd231dfd5d121b524fc3a790754 not found: ID does not exist" Nov 02 23:58:00.186236 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:00.186133 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t"] Nov 02 23:58:00.189027 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:00.189002 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e65845-openshift-default-67648bbb45-l9q8t"] Nov 02 23:58:00.713069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:00.713029 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3cff8c2-f122-4b13-a292-31bf15b6098b" path="/var/lib/kubelet/pods/a3cff8c2-f122-4b13-a292-31bf15b6098b/volumes" Nov 02 23:58:04.959868 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:04.959835 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd"] Nov 02 23:58:05.067709 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:05.067668 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh"] Nov 02 23:58:08.997131 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:08.997045 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs"] Nov 02 23:58:08.997567 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:08.997553 5672 cpu_manager.go:414] "containerMap: removing container" podUID="1663304d-daba-420d-8c7c-bd104b1ee141" containerName="revoke-keys" Nov 02 23:58:08.997615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:08.997568 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1663304d-daba-420d-8c7c-bd104b1ee141" containerName="revoke-keys" Nov 02 23:58:08.997615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:08.997587 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a3cff8c2-f122-4b13-a292-31bf15b6098b" containerName="istio-proxy" Nov 02 23:58:08.997615 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:08.997591 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3cff8c2-f122-4b13-a292-31bf15b6098b" containerName="istio-proxy" Nov 02 23:58:08.997714 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:08.997669 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="1663304d-daba-420d-8c7c-bd104b1ee141" containerName="revoke-keys" Nov 02 23:58:08.997714 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:08.997680 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3cff8c2-f122-4b13-a292-31bf15b6098b" containerName="istio-proxy" Nov 02 23:58:09.000963 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.000945 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 02 23:58:09.056657 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.056624 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e1be81\"" Nov 02 23:58:09.059943 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.059748 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs"] Nov 02 23:58:09.097547 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.097512 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e1be81-dockercfg-5ghq7\"" Nov 02 23:58:09.130018 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.129976 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdbd\" (UniqueName: \"kubernetes.io/projected/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-kube-api-access-jqdbd\") pod \"payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs\" (UID: \"0c192e7c-1b76-4d44-9b98-05f1e3c235f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 02 23:58:09.130018 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.130018 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs\" (UID: \"0c192e7c-1b76-4d44-9b98-05f1e3c235f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 02 23:58:09.148740 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.148708 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4"] Nov 02 23:58:09.153373 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.153347 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 02 23:58:09.205352 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.205318 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4"] Nov 02 23:58:09.231886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.231849 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdtk\" (UniqueName: \"kubernetes.io/projected/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-kube-api-access-vxdtk\") pod \"payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4\" (UID: \"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 02 23:58:09.232090 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.232061 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4\" (UID: \"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 02 23:58:09.232217 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.232109 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdbd\" (UniqueName: \"kubernetes.io/projected/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-kube-api-access-jqdbd\") pod \"payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs\" (UID: \"0c192e7c-1b76-4d44-9b98-05f1e3c235f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 02 23:58:09.232217 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.232197 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs\" (UID: \"0c192e7c-1b76-4d44-9b98-05f1e3c235f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 02 23:58:09.233060 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.233036 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs\" (UID: \"0c192e7c-1b76-4d44-9b98-05f1e3c235f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 02 23:58:09.285735 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.285678 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdbd\" (UniqueName: \"kubernetes.io/projected/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-kube-api-access-jqdbd\") pod \"payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs\" (UID: \"0c192e7c-1b76-4d44-9b98-05f1e3c235f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 02 23:58:09.314565 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.314537 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 02 23:58:09.333109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.333080 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4\" (UID: \"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 02 23:58:09.333249 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.333214 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdtk\" (UniqueName: \"kubernetes.io/projected/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-kube-api-access-vxdtk\") pod \"payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4\" (UID: \"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 02 23:58:09.334001 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.333979 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4\" (UID: \"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 02 23:58:09.389571 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.389508 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdtk\" (UniqueName: \"kubernetes.io/projected/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-kube-api-access-vxdtk\") pod \"payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4\" (UID: \"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 02 23:58:09.470250 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.470216 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 02 23:58:09.691185 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.691144 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs"] Nov 02 23:58:09.850575 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.850533 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4"] Nov 02 23:58:09.861629 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:58:09.861590 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a00ebb_4fec_46ac_9ba6_4f4c7b3eb16b.slice/crio-d5c4f92a4c8fb189a5c3fcc8774382701ebc80c17decf84c4f9ebfb6367e026e WatchSource:0}: Error finding container d5c4f92a4c8fb189a5c3fcc8774382701ebc80c17decf84c4f9ebfb6367e026e: Status 404 returned error can't find the container with id d5c4f92a4c8fb189a5c3fcc8774382701ebc80c17decf84c4f9ebfb6367e026e Nov 02 23:58:09.960891 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:09.960821 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" podUID="d32a755f-2f63-4522-b2d0-084792a5afbe" containerName="payload-processing" containerID="cri-o://e1e26283b0edd5084d738e50f08772635ef11e0984b8565ab0c459ad534a6936" gracePeriod=25 Nov 02 23:58:10.068302 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.068264 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" podUID="69f43db0-2fee-4fb8-bc4d-dd8aee83e903" containerName="payload-pre-processing" containerID="cri-o://c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba" gracePeriod=25 Nov 02 23:58:10.096459 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.096429 5672 generic.go:356] "Generic (PLEG): container finished" podID="d32a755f-2f63-4522-b2d0-084792a5afbe" containerID="e1e26283b0edd5084d738e50f08772635ef11e0984b8565ab0c459ad534a6936" exitCode=0 Nov 02 23:58:10.096604 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.096498 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" event={"ID":"d32a755f-2f63-4522-b2d0-084792a5afbe","Type":"ContainerDied","Data":"e1e26283b0edd5084d738e50f08772635ef11e0984b8565ab0c459ad534a6936"} Nov 02 23:58:10.098203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.098178 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" event={"ID":"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b","Type":"ContainerStarted","Data":"ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd"} Nov 02 23:58:10.098308 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.098223 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" event={"ID":"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b","Type":"ContainerStarted","Data":"d5c4f92a4c8fb189a5c3fcc8774382701ebc80c17decf84c4f9ebfb6367e026e"} Nov 02 23:58:10.098403 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.098351 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 02 23:58:10.100127 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.100103 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" event={"ID":"0c192e7c-1b76-4d44-9b98-05f1e3c235f7","Type":"ContainerStarted","Data":"a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7"} Nov 02 23:58:10.100276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.100157 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" event={"ID":"0c192e7c-1b76-4d44-9b98-05f1e3c235f7","Type":"ContainerStarted","Data":"a77c8fca1c85fb85d8f8e35aec38f91b5785293fd958ed35bfc1764531ea5669"} Nov 02 23:58:10.100276 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.100183 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 02 23:58:10.210944 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.208759 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" podStartSLOduration=1.208741631 podStartE2EDuration="1.208741631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:58:10.208272606 +0000 UTC m=+3883.615584446" watchObservedRunningTime="2026-11-02 23:58:10.208741631 +0000 UTC m=+3883.616053463" Nov 02 23:58:10.215955 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.215931 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:58:10.248161 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.248104 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32a755f-2f63-4522-b2d0-084792a5afbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32a755f-2f63-4522-b2d0-084792a5afbe-plugins-config-volume\") pod \"d32a755f-2f63-4522-b2d0-084792a5afbe\" (UID: \"d32a755f-2f63-4522-b2d0-084792a5afbe\") " Nov 02 23:58:10.248334 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.248318 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32a755f-2f63-4522-b2d0-084792a5afbe-kube-api-access-vvtbz\" (UniqueName: \"kubernetes.io/projected/d32a755f-2f63-4522-b2d0-084792a5afbe-kube-api-access-vvtbz\") pod \"d32a755f-2f63-4522-b2d0-084792a5afbe\" (UID: \"d32a755f-2f63-4522-b2d0-084792a5afbe\") " Nov 02 23:58:10.249246 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.249190 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32a755f-2f63-4522-b2d0-084792a5afbe-plugins-config-volume" pod "d32a755f-2f63-4522-b2d0-084792a5afbe" (UID: "d32a755f-2f63-4522-b2d0-084792a5afbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:10.251895 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.251870 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32a755f-2f63-4522-b2d0-084792a5afbe-kube-api-access-vvtbz" pod "d32a755f-2f63-4522-b2d0-084792a5afbe" (UID: "d32a755f-2f63-4522-b2d0-084792a5afbe"). InnerVolumeSpecName "kube-api-access-vvtbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:10.318043 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.317986 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" podStartSLOduration=2.317970481 podStartE2EDuration="2.317970481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:58:10.315809475 +0000 UTC m=+3883.723121302" watchObservedRunningTime="2026-11-02 23:58:10.317970481 +0000 UTC m=+3883.725282328" Nov 02 23:58:10.349374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.349349 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvtbz\" (UniqueName: \"kubernetes.io/projected/d32a755f-2f63-4522-b2d0-084792a5afbe-kube-api-access-vvtbz\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:58:10.349374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.349372 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32a755f-2f63-4522-b2d0-084792a5afbe-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:58:10.352620 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.352593 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:58:10.450107 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.450072 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-kube-api-access-4v7hz\" (UniqueName: \"kubernetes.io/projected/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-kube-api-access-4v7hz\") pod \"69f43db0-2fee-4fb8-bc4d-dd8aee83e903\" (UID: \"69f43db0-2fee-4fb8-bc4d-dd8aee83e903\") " Nov 02 23:58:10.450327 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.450299 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-plugins-config-volume\") pod \"69f43db0-2fee-4fb8-bc4d-dd8aee83e903\" (UID: \"69f43db0-2fee-4fb8-bc4d-dd8aee83e903\") " Nov 02 23:58:10.450957 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.450923 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-plugins-config-volume" pod "69f43db0-2fee-4fb8-bc4d-dd8aee83e903" (UID: "69f43db0-2fee-4fb8-bc4d-dd8aee83e903"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:58:10.464631 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.464602 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-kube-api-access-4v7hz" pod "69f43db0-2fee-4fb8-bc4d-dd8aee83e903" (UID: "69f43db0-2fee-4fb8-bc4d-dd8aee83e903"). InnerVolumeSpecName "kube-api-access-4v7hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:58:10.551635 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.551598 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v7hz\" (UniqueName: \"kubernetes.io/projected/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-kube-api-access-4v7hz\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:58:10.551635 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:10.551636 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f43db0-2fee-4fb8-bc4d-dd8aee83e903-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:58:11.108085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.108005 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" Nov 02 23:58:11.108085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.108028 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd" event={"ID":"d32a755f-2f63-4522-b2d0-084792a5afbe","Type":"ContainerDied","Data":"75e67bc5e049b1c6c67b0d3c1175c5b45994ed2f199499ee8ddfc916b2486ab1"} Nov 02 23:58:11.108626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.108091 5672 scope.go:122] "RemoveContainer" containerID="e1e26283b0edd5084d738e50f08772635ef11e0984b8565ab0c459ad534a6936" Nov 02 23:58:11.109658 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.109635 5672 generic.go:356] "Generic (PLEG): container finished" podID="69f43db0-2fee-4fb8-bc4d-dd8aee83e903" containerID="c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba" exitCode=0 Nov 02 23:58:11.109750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.109669 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" Nov 02 23:58:11.109750 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.109715 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" event={"ID":"69f43db0-2fee-4fb8-bc4d-dd8aee83e903","Type":"ContainerDied","Data":"c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba"} Nov 02 23:58:11.109832 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.109758 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh" event={"ID":"69f43db0-2fee-4fb8-bc4d-dd8aee83e903","Type":"ContainerDied","Data":"1f1dd7c86b4aa35949a9659db66145cc11955618ec4f19b700ab5bb4962f486f"} Nov 02 23:58:11.118483 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.118459 5672 scope.go:122] "RemoveContainer" containerID="c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba" Nov 02 23:58:11.127217 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.127199 5672 scope.go:122] "RemoveContainer" containerID="c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba" Nov 02 23:58:11.127523 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:58:11.127500 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba\": container with ID starting with c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba not found: ID does not exist" containerID="c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba" Nov 02 23:58:11.127622 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.127536 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba"} err="failed to get container status \"c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba\": rpc error: code = NotFound desc = could not find container \"c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba\": container with ID starting with c5a250f7efbbe6885847efb49f4d421712572527923379ce7b583bb27a6f25ba not found: ID does not exist" Nov 02 23:58:11.263339 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.263305 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd"] Nov 02 23:58:11.267318 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.267291 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a8a1dd-6cf569878c-f87hd"] Nov 02 23:58:11.365652 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.365578 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh"] Nov 02 23:58:11.372044 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:11.372018 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a8a1dd-fdd99c687-gbjrh"] Nov 02 23:58:12.713406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:12.713374 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f43db0-2fee-4fb8-bc4d-dd8aee83e903" path="/var/lib/kubelet/pods/69f43db0-2fee-4fb8-bc4d-dd8aee83e903/volumes" Nov 02 23:58:12.713945 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:12.713926 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32a755f-2f63-4522-b2d0-084792a5afbe" path="/var/lib/kubelet/pods/d32a755f-2f63-4522-b2d0-084792a5afbe/volumes" Nov 02 23:58:16.205626 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:16.205592 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:58:21.112747 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:21.112714 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 02 23:58:21.113177 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:21.112786 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 02 23:58:26.020857 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:58:26.020817 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:58:29.296931 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:58:29.296900 5672 scope.go:122] "RemoveContainer" containerID="11ce77f668fff1b2e50f929ce172b13a96d1102cba5f843e5cc592f1f573eaa8" Nov 02 23:58:56.020522 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:58:56.020481 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:59:14.061406 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:14.061363 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 02 23:59:18.039026 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.038991 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z"] Nov 02 23:59:18.039566 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.039551 5672 cpu_manager.go:414] "containerMap: removing container" podUID="69f43db0-2fee-4fb8-bc4d-dd8aee83e903" containerName="payload-pre-processing" Nov 02 23:59:18.039630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.039568 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f43db0-2fee-4fb8-bc4d-dd8aee83e903" containerName="payload-pre-processing" Nov 02 23:59:18.039630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.039613 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d32a755f-2f63-4522-b2d0-084792a5afbe" containerName="payload-processing" Nov 02 23:59:18.039630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.039619 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32a755f-2f63-4522-b2d0-084792a5afbe" containerName="payload-processing" Nov 02 23:59:18.039726 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.039703 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f43db0-2fee-4fb8-bc4d-dd8aee83e903" containerName="payload-pre-processing" Nov 02 23:59:18.039726 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.039716 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32a755f-2f63-4522-b2d0-084792a5afbe" containerName="payload-processing" Nov 02 23:59:18.043273 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.043251 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.099179 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.096501 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-68d89f-openshift-default-dockercfg-6mldd\"" Nov 02 23:59:18.112169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.111797 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-socket\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.112169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.111894 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-token\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.112169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.111939 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-credential-socket\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.112169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.111964 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-certs\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.112169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.111996 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-data\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.112169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.112032 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-podinfo\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.112169 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.112059 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnz4l\" (UniqueName: \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-kube-api-access-nnz4l\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.112677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.112210 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-envoy\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.112677 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.112279 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istiod-ca-cert\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.146295 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.146261 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z"] Nov 02 23:59:18.213238 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213204 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istiod-ca-cert\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.213416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213246 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-socket\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.213416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213314 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-token\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.213416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213350 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-credential-socket\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.213416 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213374 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-certs\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.213561 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213422 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-data\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.213788 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213762 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-podinfo\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.213877 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213808 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnz4l\" (UniqueName: \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-kube-api-access-nnz4l\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.213877 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213810 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-data\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.213970 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213903 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-certs\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.214031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213901 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-credential-socket\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.214031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213766 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-socket\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.214031 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.214012 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istiod-ca-cert\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.214164 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.213977 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-envoy\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.216003 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.215984 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-envoy\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.216203 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.216184 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-podinfo\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.267686 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.267647 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-token\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.267862 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.267714 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnz4l\" (UniqueName: \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-kube-api-access-nnz4l\") pod \"e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.360347 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.360253 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:18.836101 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.836064 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z"] Nov 02 23:59:18.873978 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:59:18.873952 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e85ff01_0bf8_4f3a_92ed_ac012f54e8d0.slice/crio-e94d51940c08d2713c232891793b9228b13b95aa2b942d906e06b8cdfcbe23e5 WatchSource:0}: Error finding container e94d51940c08d2713c232891793b9228b13b95aa2b942d906e06b8cdfcbe23e5: Status 404 returned error can't find the container with id e94d51940c08d2713c232891793b9228b13b95aa2b942d906e06b8cdfcbe23e5 Nov 02 23:59:18.877003 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.876970 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:18.877085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.877043 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:18.877085 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:18.877071 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:19.547920 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:19.547880 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" event={"ID":"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0","Type":"ContainerStarted","Data":"5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe"} Nov 02 23:59:19.548451 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:19.547938 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" event={"ID":"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0","Type":"ContainerStarted","Data":"e94d51940c08d2713c232891793b9228b13b95aa2b942d906e06b8cdfcbe23e5"} Nov 02 23:59:19.658958 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:19.658904 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" podStartSLOduration=2.658890298 podStartE2EDuration="2.658890298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:19.655832042 +0000 UTC m=+3953.063143871" watchObservedRunningTime="2026-11-02 23:59:19.658890298 +0000 UTC m=+3953.066202127" Nov 02 23:59:20.360380 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:20.360347 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:20.366726 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:20.366689 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:20.554699 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:20.554668 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:20.555576 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:20.555557 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 02 23:59:26.040621 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:59:26.040583 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:59:26.056684 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.056650 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht"] Nov 02 23:59:26.061997 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.061966 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 02 23:59:26.110423 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.110393 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht"] Nov 02 23:59:26.153515 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.153484 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-68d89f-dockercfg-ngvj5\"" Nov 02 23:59:26.153721 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.153702 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-68d89f\"" Nov 02 23:59:26.206942 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.206909 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht\" (UID: \"e73cece6-ad1e-48af-9350-c1cd2d0d05f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 02 23:59:26.207109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.207069 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn72f\" (UniqueName: \"kubernetes.io/projected/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-kube-api-access-wn72f\") pod \"payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht\" (UID: \"e73cece6-ad1e-48af-9350-c1cd2d0d05f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 02 23:59:26.308499 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.308425 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht\" (UID: \"e73cece6-ad1e-48af-9350-c1cd2d0d05f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 02 23:59:26.308640 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.308516 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn72f\" (UniqueName: \"kubernetes.io/projected/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-kube-api-access-wn72f\") pod \"payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht\" (UID: \"e73cece6-ad1e-48af-9350-c1cd2d0d05f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 02 23:59:26.309114 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.309094 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht\" (UID: \"e73cece6-ad1e-48af-9350-c1cd2d0d05f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 02 23:59:26.362343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.362306 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn72f\" (UniqueName: \"kubernetes.io/projected/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-kube-api-access-wn72f\") pod \"payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht\" (UID: \"e73cece6-ad1e-48af-9350-c1cd2d0d05f4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 02 23:59:26.379668 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.379642 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 02 23:59:26.752837 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:26.752806 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht"] Nov 02 23:59:26.754726 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:59:26.754698 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73cece6_ad1e_48af_9350_c1cd2d0d05f4.slice/crio-5b7b3ce6f2b536422d8ecabf7db9c1e68f572c8c4771a1c98a9ce73df3234f16 WatchSource:0}: Error finding container 5b7b3ce6f2b536422d8ecabf7db9c1e68f572c8c4771a1c98a9ce73df3234f16: Status 404 returned error can't find the container with id 5b7b3ce6f2b536422d8ecabf7db9c1e68f572c8c4771a1c98a9ce73df3234f16 Nov 02 23:59:27.601989 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:27.601945 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" event={"ID":"e73cece6-ad1e-48af-9350-c1cd2d0d05f4","Type":"ContainerStarted","Data":"ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584"} Nov 02 23:59:27.602503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:27.602013 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" event={"ID":"e73cece6-ad1e-48af-9350-c1cd2d0d05f4","Type":"ContainerStarted","Data":"5b7b3ce6f2b536422d8ecabf7db9c1e68f572c8c4771a1c98a9ce73df3234f16"} Nov 02 23:59:27.602503 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:27.602040 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 02 23:59:27.711601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:27.711546 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" podStartSLOduration=1.7115269199999998 podStartE2EDuration="1.71152692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:27.708979436 +0000 UTC m=+3961.116291264" watchObservedRunningTime="2026-11-02 23:59:27.71152692 +0000 UTC m=+3961.118838750" Nov 02 23:59:28.054771 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.051379 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx"] Nov 02 23:59:28.058491 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.058459 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:28.106933 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.106906 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx"] Nov 02 23:59:28.107607 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.107562 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 23:59:28.150109 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.150079 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 02 23:59:28.150244 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.150137 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 23:59:28.150244 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.150200 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 23:59:28.234720 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.234687 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348f7197-qlljx\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:28.234878 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.234745 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348f7197-qlljx\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:28.234932 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.234883 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz66s\" (UniqueName: \"kubernetes.io/projected/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-kube-api-access-pz66s\") pod \"maas-api-revoke-sub-348f7197-qlljx\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:28.336424 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.336335 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348f7197-qlljx\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:28.336600 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.336477 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz66s\" (UniqueName: \"kubernetes.io/projected/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-kube-api-access-pz66s\") pod \"maas-api-revoke-sub-348f7197-qlljx\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:28.336600 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.336576 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348f7197-qlljx\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:28.337088 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.337057 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348f7197-qlljx\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:28.339116 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.339090 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348f7197-qlljx\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:28.389630 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.389596 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz66s\" (UniqueName: \"kubernetes.io/projected/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-kube-api-access-pz66s\") pod \"maas-api-revoke-sub-348f7197-qlljx\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:28.675029 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:28.674949 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:29.043119 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:29.043090 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx"] Nov 02 23:59:29.045905 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:59:29.045859 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f24a9a_156c_4c98_b9b9_b5caeea3cee0.slice/crio-6f52f0b96940641c053cdcbc32048c858582ecd47fb7c75201592538b425a664 WatchSource:0}: Error finding container 6f52f0b96940641c053cdcbc32048c858582ecd47fb7c75201592538b425a664: Status 404 returned error can't find the container with id 6f52f0b96940641c053cdcbc32048c858582ecd47fb7c75201592538b425a664 Nov 02 23:59:29.616214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:29.616125 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" event={"ID":"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0","Type":"ContainerStarted","Data":"061a6111edbd271a83560e1b3d36e5419830c79472c24c524c982af78d4d4994"} Nov 02 23:59:29.616214 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:29.616202 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" event={"ID":"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0","Type":"ContainerStarted","Data":"6f52f0b96940641c053cdcbc32048c858582ecd47fb7c75201592538b425a664"} Nov 02 23:59:29.724030 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:29.723977 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" podStartSLOduration=1.723962913 podStartE2EDuration="1.723962913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:29.721326892 +0000 UTC m=+3963.128638723" watchObservedRunningTime="2026-11-02 23:59:29.723962913 +0000 UTC m=+3963.131274742" Nov 02 23:59:30.624011 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:30.623975 5672 generic.go:356] "Generic (PLEG): container finished" podID="e7f24a9a-156c-4c98-b9b9-b5caeea3cee0" containerID="061a6111edbd271a83560e1b3d36e5419830c79472c24c524c982af78d4d4994" exitCode=0 Nov 02 23:59:30.624253 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:30.624033 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" event={"ID":"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0","Type":"ContainerDied","Data":"061a6111edbd271a83560e1b3d36e5419830c79472c24c524c982af78d4d4994"} Nov 02 23:59:31.789450 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:31.789425 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:31.982900 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:31.982873 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-kube-api-access-pz66s\" (UniqueName: \"kubernetes.io/projected/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-kube-api-access-pz66s\") pod \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " Nov 02 23:59:31.983071 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:31.982938 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-openshift-ca\") pod \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " Nov 02 23:59:31.983071 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:31.982962 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-xks-ca\") pod \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\" (UID: \"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0\") " Nov 02 23:59:31.983374 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:31.983341 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-openshift-ca" pod "e7f24a9a-156c-4c98-b9b9-b5caeea3cee0" (UID: "e7f24a9a-156c-4c98-b9b9-b5caeea3cee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 23:59:31.985017 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:31.984987 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-xks-ca" pod "e7f24a9a-156c-4c98-b9b9-b5caeea3cee0" (UID: "e7f24a9a-156c-4c98-b9b9-b5caeea3cee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 23:59:31.985121 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:31.985059 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-kube-api-access-pz66s" pod "e7f24a9a-156c-4c98-b9b9-b5caeea3cee0" (UID: "e7f24a9a-156c-4c98-b9b9-b5caeea3cee0"). InnerVolumeSpecName "kube-api-access-pz66s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 23:59:32.084681 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:32.084652 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz66s\" (UniqueName: \"kubernetes.io/projected/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-kube-api-access-pz66s\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:32.084681 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:32.084679 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:32.084867 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:32.084689 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 23:59:32.641825 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:32.641794 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" Nov 02 23:59:32.642040 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:32.641840 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx" event={"ID":"e7f24a9a-156c-4c98-b9b9-b5caeea3cee0","Type":"ContainerDied","Data":"6f52f0b96940641c053cdcbc32048c858582ecd47fb7c75201592538b425a664"} Nov 02 23:59:32.642040 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:32.641900 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f52f0b96940641c053cdcbc32048c858582ecd47fb7c75201592538b425a664" Nov 02 23:59:38.611940 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:38.611869 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 02 23:59:39.067256 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.067214 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns"] Nov 02 23:59:39.067751 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.067739 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e7f24a9a-156c-4c98-b9b9-b5caeea3cee0" containerName="revoke-keys" Nov 02 23:59:39.067813 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.067752 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f24a9a-156c-4c98-b9b9-b5caeea3cee0" containerName="revoke-keys" Nov 02 23:59:39.067858 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.067848 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f24a9a-156c-4c98-b9b9-b5caeea3cee0" containerName="revoke-keys" Nov 02 23:59:39.071123 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.071105 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.156895 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.156854 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.157078 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.157025 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.157141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.157085 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.157141 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.157120 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.157264 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.157222 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4hd\" (UniqueName: \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-kube-api-access-fk4hd\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.157322 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.157270 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.157377 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.157322 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.157430 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.157403 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.157496 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.157483 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84720bad-6f3c-48da-9fec-8dc56a7de315-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.256482 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.256447 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns"] Nov 02 23:59:39.258995 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.258952 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.259178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.259033 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.259178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.259065 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.259178 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.259127 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84720bad-6f3c-48da-9fec-8dc56a7de315-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.259387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.259200 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.259387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.259299 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.259387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.259329 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.259387 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.259356 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.259655 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.259392 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4hd\" (UniqueName: \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-kube-api-access-fk4hd\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.260167 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.260113 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.260507 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.260487 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.260867 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.260579 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84720bad-6f3c-48da-9fec-8dc56a7de315-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.260993 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.260705 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.261143 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.260817 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.266335 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.262535 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.270687 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.270659 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.313732 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.313705 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4hd\" (UniqueName: \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-kube-api-access-fk4hd\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.313866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.313707 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-57kns\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.384480 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.384415 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:39.805161 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.805119 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns"] Nov 02 23:59:39.806413 ip-10-0-137-4 kubenswrapper[5672]: W1102 23:59:39.806368 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84720bad_6f3c_48da_9fec_8dc56a7de315.slice/crio-6a3076881ee9c6879e54413bf9ea21f919f207e8ba0e84c5907f2708bedf30cc WatchSource:0}: Error finding container 6a3076881ee9c6879e54413bf9ea21f919f207e8ba0e84c5907f2708bedf30cc: Status 404 returned error can't find the container with id 6a3076881ee9c6879e54413bf9ea21f919f207e8ba0e84c5907f2708bedf30cc Nov 02 23:59:39.808968 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.808934 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:39.809069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.809014 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:39.809069 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:39.809057 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 23:59:40.694192 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:40.694125 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" event={"ID":"84720bad-6f3c-48da-9fec-8dc56a7de315","Type":"ContainerStarted","Data":"dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c"} Nov 02 23:59:40.694368 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:40.694198 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" event={"ID":"84720bad-6f3c-48da-9fec-8dc56a7de315","Type":"ContainerStarted","Data":"6a3076881ee9c6879e54413bf9ea21f919f207e8ba0e84c5907f2708bedf30cc"} Nov 02 23:59:41.384839 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:41.384805 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:41.386287 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:41.386261 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-57kns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 02 23:59:41.386471 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:41.386328 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 02 23:59:42.385572 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:42.385537 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-57kns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 02 23:59:42.386082 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:42.385597 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 02 23:59:43.385601 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:43.385562 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-57kns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 02 23:59:43.385994 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:43.385626 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 02 23:59:44.385665 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:44.385620 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-57kns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 02 23:59:44.386189 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:44.385676 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 02 23:59:45.385166 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:45.385116 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-57kns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 02 23:59:45.385343 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:45.385190 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 02 23:59:46.385274 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:46.385234 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-57kns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 02 23:59:46.385800 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:46.385295 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 02 23:59:47.387340 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:47.387301 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-57kns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 02 23:59:47.387917 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:47.387879 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 02 23:59:48.498393 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:48.498332 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:48.498866 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:48.498768 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:48.499886 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:48.499861 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 02 23:59:48.701624 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:48.701565 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" podStartSLOduration=9.701551233 podStartE2EDuration="9.701551233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:40.902801197 +0000 UTC m=+3974.310113026" watchObservedRunningTime="2026-11-02 23:59:48.701551233 +0000 UTC m=+3982.108863063" Nov 02 23:59:56.020264 ip-10-0-137-4 kubenswrapper[5672]: E1102 23:59:56.020228 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 02 23:59:58.478741 ip-10-0-137-4 kubenswrapper[5672]: I1102 23:59:58.478707 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:00:00.171384 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.171338 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b"] Nov 03 00:00:00.177322 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.177295 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25"] Nov 03 00:00:00.177504 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.177406 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" Nov 03 00:00:00.181427 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.181405 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b"] Nov 03 00:00:00.181655 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.181639 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" Nov 03 00:00:00.187813 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.186680 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" Nov 03 00:00:00.223202 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.216574 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm"] Nov 03 00:00:00.223202 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.221239 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf"] Nov 03 00:00:00.223202 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.221344 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" Nov 03 00:00:00.226257 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.226119 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj"] Nov 03 00:00:00.229195 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.228459 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" Nov 03 00:00:00.231853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.231831 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm"] Nov 03 00:00:00.235168 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.234915 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" Nov 03 00:00:00.235943 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.235924 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" Nov 03 00:00:00.242469 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.242442 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz"] Nov 03 00:00:00.245269 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.245241 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6fx8\" (UniqueName: \"kubernetes.io/projected/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-kube-api-access-v6fx8\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b\" (UID: \"c6b2d887-1c8e-4028-98be-5c90ee7bb36d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" Nov 03 00:00:00.245482 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.245456 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b\" (UID: \"c6b2d887-1c8e-4028-98be-5c90ee7bb36d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" Nov 03 00:00:00.245599 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.245534 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a2752d-fd8e-4438-87b1-ade1d475dc9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b\" (UID: \"79a2752d-fd8e-4438-87b1-ade1d475dc9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" Nov 03 00:00:00.245670 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.245653 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv68l\" (UniqueName: \"kubernetes.io/projected/79a2752d-fd8e-4438-87b1-ade1d475dc9a-kube-api-access-cv68l\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b\" (UID: \"79a2752d-fd8e-4438-87b1-ade1d475dc9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" Nov 03 00:00:00.245824 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.245808 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513fd003-68d9-4e05-bf59-3c3369882a4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25\" (UID: \"513fd003-68d9-4e05-bf59-3c3369882a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" Nov 03 00:00:00.245952 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.245935 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwzzf\" (UniqueName: \"kubernetes.io/projected/513fd003-68d9-4e05-bf59-3c3369882a4b-kube-api-access-xwzzf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25\" (UID: \"513fd003-68d9-4e05-bf59-3c3369882a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" Nov 03 00:00:00.247553 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.247531 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg"] Nov 03 00:00:00.248437 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.248417 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" Nov 03 00:00:00.255956 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.255933 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" Nov 03 00:00:00.272247 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.272207 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4"] Nov 03 00:00:00.278600 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.278575 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" Nov 03 00:00:00.289532 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.289514 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25"] Nov 03 00:00:00.299308 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.294915 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b"] Nov 03 00:00:00.310452 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.308998 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b"] Nov 03 00:00:00.341171 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.337106 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:00:00.341171 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.337248 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:00:00.341171 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.337285 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm"] Nov 03 00:00:00.341171 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.340763 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:00:00.345197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.343578 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj"] Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347114 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3fac2-f23d-49a6-b27a-b395e1722d75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz\" (UID: \"2cf3fac2-f23d-49a6-b27a-b395e1722d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347182 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv68l\" (UniqueName: \"kubernetes.io/projected/79a2752d-fd8e-4438-87b1-ade1d475dc9a-kube-api-access-cv68l\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b\" (UID: \"79a2752d-fd8e-4438-87b1-ade1d475dc9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347234 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513fd003-68d9-4e05-bf59-3c3369882a4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25\" (UID: \"513fd003-68d9-4e05-bf59-3c3369882a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347272 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzzf\" (UniqueName: \"kubernetes.io/projected/513fd003-68d9-4e05-bf59-3c3369882a4b-kube-api-access-xwzzf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25\" (UID: \"513fd003-68d9-4e05-bf59-3c3369882a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347309 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295e9734-7b84-4a80-a5d7-11d68964f215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm\" (UID: \"295e9734-7b84-4a80-a5d7-11d68964f215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347348 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmltx\" (UniqueName: \"kubernetes.io/projected/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-kube-api-access-vmltx\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg\" (UID: \"8c2c9c15-097d-4fc2-a471-deb4c54dad7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347378 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5610bb86-f95e-4ee3-81db-63edc417a640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj\" (UID: \"5610bb86-f95e-4ee3-81db-63edc417a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347455 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6fx8\" (UniqueName: \"kubernetes.io/projected/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-kube-api-access-v6fx8\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b\" (UID: \"c6b2d887-1c8e-4028-98be-5c90ee7bb36d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347506 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hczwr\" (UniqueName: \"kubernetes.io/projected/7131cd26-6c33-4d0f-a629-d390bc70b0c2-kube-api-access-hczwr\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm\" (UID: \"7131cd26-6c33-4d0f-a629-d390bc70b0c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347558 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b\" (UID: \"c6b2d887-1c8e-4028-98be-5c90ee7bb36d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347597 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtjp\" (UniqueName: \"kubernetes.io/projected/2cf3fac2-f23d-49a6-b27a-b395e1722d75-kube-api-access-7qtjp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz\" (UID: \"2cf3fac2-f23d-49a6-b27a-b395e1722d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347635 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxsjx\" (UniqueName: \"kubernetes.io/projected/295e9734-7b84-4a80-a5d7-11d68964f215-kube-api-access-hxsjx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm\" (UID: \"295e9734-7b84-4a80-a5d7-11d68964f215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347666 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkcpj\" (UniqueName: \"kubernetes.io/projected/85f4c6ca-d293-4bfe-b736-888f35359964-kube-api-access-zkcpj\") pod \"maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4\" (UID: \"85f4c6ca-d293-4bfe-b736-888f35359964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347734 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f4c6ca-d293-4bfe-b736-888f35359964-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4\" (UID: \"85f4c6ca-d293-4bfe-b736-888f35359964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347806 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a2752d-fd8e-4438-87b1-ade1d475dc9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b\" (UID: \"79a2752d-fd8e-4438-87b1-ade1d475dc9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347878 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2fl\" (UniqueName: \"kubernetes.io/projected/5610bb86-f95e-4ee3-81db-63edc417a640-kube-api-access-ns2fl\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj\" (UID: \"5610bb86-f95e-4ee3-81db-63edc417a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347915 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7131cd26-6c33-4d0f-a629-d390bc70b0c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm\" (UID: \"7131cd26-6c33-4d0f-a629-d390bc70b0c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.347969 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5dee99d-d0f4-4e91-abdf-b21758d550d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf\" (UID: \"d5dee99d-d0f4-4e91-abdf-b21758d550d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.348027 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg\" (UID: \"8c2c9c15-097d-4fc2-a471-deb4c54dad7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" Nov 03 00:00:00.349283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.348119 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqlw\" (UniqueName: \"kubernetes.io/projected/d5dee99d-d0f4-4e91-abdf-b21758d550d4-kube-api-access-vsqlw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf\" (UID: \"d5dee99d-d0f4-4e91-abdf-b21758d550d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" Nov 03 00:00:00.350617 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.350559 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b\" (UID: \"c6b2d887-1c8e-4028-98be-5c90ee7bb36d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" Nov 03 00:00:00.416786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.351344 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a2752d-fd8e-4438-87b1-ade1d475dc9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b\" (UID: \"79a2752d-fd8e-4438-87b1-ade1d475dc9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" Nov 03 00:00:00.416786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.352871 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513fd003-68d9-4e05-bf59-3c3369882a4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25\" (UID: \"513fd003-68d9-4e05-bf59-3c3369882a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" Nov 03 00:00:00.416786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.356177 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz"] Nov 03 00:00:00.416786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.390905 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg"] Nov 03 00:00:00.416786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.396275 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4"] Nov 03 00:00:00.434994 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.434888 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6fx8\" (UniqueName: \"kubernetes.io/projected/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-kube-api-access-v6fx8\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b\" (UID: \"c6b2d887-1c8e-4028-98be-5c90ee7bb36d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.454612 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqlw\" (UniqueName: \"kubernetes.io/projected/d5dee99d-d0f4-4e91-abdf-b21758d550d4-kube-api-access-vsqlw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf\" (UID: \"d5dee99d-d0f4-4e91-abdf-b21758d550d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.454683 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3fac2-f23d-49a6-b27a-b395e1722d75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz\" (UID: \"2cf3fac2-f23d-49a6-b27a-b395e1722d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.455746 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295e9734-7b84-4a80-a5d7-11d68964f215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm\" (UID: \"295e9734-7b84-4a80-a5d7-11d68964f215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.455823 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmltx\" (UniqueName: \"kubernetes.io/projected/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-kube-api-access-vmltx\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg\" (UID: \"8c2c9c15-097d-4fc2-a471-deb4c54dad7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.455861 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5610bb86-f95e-4ee3-81db-63edc417a640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj\" (UID: \"5610bb86-f95e-4ee3-81db-63edc417a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.455960 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hczwr\" (UniqueName: \"kubernetes.io/projected/7131cd26-6c33-4d0f-a629-d390bc70b0c2-kube-api-access-hczwr\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm\" (UID: \"7131cd26-6c33-4d0f-a629-d390bc70b0c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.456018 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtjp\" (UniqueName: \"kubernetes.io/projected/2cf3fac2-f23d-49a6-b27a-b395e1722d75-kube-api-access-7qtjp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz\" (UID: \"2cf3fac2-f23d-49a6-b27a-b395e1722d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.456063 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxsjx\" (UniqueName: \"kubernetes.io/projected/295e9734-7b84-4a80-a5d7-11d68964f215-kube-api-access-hxsjx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm\" (UID: \"295e9734-7b84-4a80-a5d7-11d68964f215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.456096 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkcpj\" (UniqueName: \"kubernetes.io/projected/85f4c6ca-d293-4bfe-b736-888f35359964-kube-api-access-zkcpj\") pod \"maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4\" (UID: \"85f4c6ca-d293-4bfe-b736-888f35359964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.456139 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f4c6ca-d293-4bfe-b736-888f35359964-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4\" (UID: \"85f4c6ca-d293-4bfe-b736-888f35359964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.456231 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2fl\" (UniqueName: \"kubernetes.io/projected/5610bb86-f95e-4ee3-81db-63edc417a640-kube-api-access-ns2fl\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj\" (UID: \"5610bb86-f95e-4ee3-81db-63edc417a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.456274 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7131cd26-6c33-4d0f-a629-d390bc70b0c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm\" (UID: \"7131cd26-6c33-4d0f-a629-d390bc70b0c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" Nov 03 00:00:00.457186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.457188 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5dee99d-d0f4-4e91-abdf-b21758d550d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf\" (UID: \"d5dee99d-d0f4-4e91-abdf-b21758d550d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" Nov 03 00:00:00.458048 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.457260 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg\" (UID: \"8c2c9c15-097d-4fc2-a471-deb4c54dad7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" Nov 03 00:00:00.461166 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.458238 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg\" (UID: \"8c2c9c15-097d-4fc2-a471-deb4c54dad7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" Nov 03 00:00:00.461166 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.459998 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3fac2-f23d-49a6-b27a-b395e1722d75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz\" (UID: \"2cf3fac2-f23d-49a6-b27a-b395e1722d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" Nov 03 00:00:00.461166 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.460699 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295e9734-7b84-4a80-a5d7-11d68964f215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm\" (UID: \"295e9734-7b84-4a80-a5d7-11d68964f215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" Nov 03 00:00:00.462166 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.461656 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5610bb86-f95e-4ee3-81db-63edc417a640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj\" (UID: \"5610bb86-f95e-4ee3-81db-63edc417a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" Nov 03 00:00:00.464538 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.463469 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f4c6ca-d293-4bfe-b736-888f35359964-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4\" (UID: \"85f4c6ca-d293-4bfe-b736-888f35359964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" Nov 03 00:00:00.465874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.464885 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7131cd26-6c33-4d0f-a629-d390bc70b0c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm\" (UID: \"7131cd26-6c33-4d0f-a629-d390bc70b0c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" Nov 03 00:00:00.465874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.464911 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv68l\" (UniqueName: \"kubernetes.io/projected/79a2752d-fd8e-4438-87b1-ade1d475dc9a-kube-api-access-cv68l\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b\" (UID: \"79a2752d-fd8e-4438-87b1-ade1d475dc9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" Nov 03 00:00:00.465874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.465634 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5dee99d-d0f4-4e91-abdf-b21758d550d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf\" (UID: \"d5dee99d-d0f4-4e91-abdf-b21758d550d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" Nov 03 00:00:00.473961 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.473653 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzzf\" (UniqueName: \"kubernetes.io/projected/513fd003-68d9-4e05-bf59-3c3369882a4b-kube-api-access-xwzzf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25\" (UID: \"513fd003-68d9-4e05-bf59-3c3369882a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" Nov 03 00:00:00.480103 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.480078 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm"] Nov 03 00:00:00.483558 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.483218 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf"] Nov 03 00:00:00.515414 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.515376 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hczwr\" (UniqueName: \"kubernetes.io/projected/7131cd26-6c33-4d0f-a629-d390bc70b0c2-kube-api-access-hczwr\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm\" (UID: \"7131cd26-6c33-4d0f-a629-d390bc70b0c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" Nov 03 00:00:00.515740 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.515713 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmltx\" (UniqueName: \"kubernetes.io/projected/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-kube-api-access-vmltx\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg\" (UID: \"8c2c9c15-097d-4fc2-a471-deb4c54dad7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" Nov 03 00:00:00.515988 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.515968 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqlw\" (UniqueName: \"kubernetes.io/projected/d5dee99d-d0f4-4e91-abdf-b21758d550d4-kube-api-access-vsqlw\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf\" (UID: \"d5dee99d-d0f4-4e91-abdf-b21758d550d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" Nov 03 00:00:00.517405 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.517346 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxsjx\" (UniqueName: \"kubernetes.io/projected/295e9734-7b84-4a80-a5d7-11d68964f215-kube-api-access-hxsjx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm\" (UID: \"295e9734-7b84-4a80-a5d7-11d68964f215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" Nov 03 00:00:00.517963 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.517939 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkcpj\" (UniqueName: \"kubernetes.io/projected/85f4c6ca-d293-4bfe-b736-888f35359964-kube-api-access-zkcpj\") pod \"maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4\" (UID: \"85f4c6ca-d293-4bfe-b736-888f35359964\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" Nov 03 00:00:00.518082 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.518064 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtjp\" (UniqueName: \"kubernetes.io/projected/2cf3fac2-f23d-49a6-b27a-b395e1722d75-kube-api-access-7qtjp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz\" (UID: \"2cf3fac2-f23d-49a6-b27a-b395e1722d75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" Nov 03 00:00:00.518271 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.518246 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2fl\" (UniqueName: \"kubernetes.io/projected/5610bb86-f95e-4ee3-81db-63edc417a640-kube-api-access-ns2fl\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj\" (UID: \"5610bb86-f95e-4ee3-81db-63edc417a640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" Nov 03 00:00:00.539111 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.539083 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" Nov 03 00:00:00.554528 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.554289 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" Nov 03 00:00:00.570716 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.567124 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" Nov 03 00:00:00.584200 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.584052 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" Nov 03 00:00:00.592271 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.592084 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" Nov 03 00:00:00.615927 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.615713 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" Nov 03 00:00:00.695140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.694850 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" Nov 03 00:00:00.713188 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.713090 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" Nov 03 00:00:00.780455 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.780370 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" Nov 03 00:00:00.807870 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.803739 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" Nov 03 00:00:00.970757 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:00.970070 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b"] Nov 03 00:00:00.985554 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:00:00.984072 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a2752d_fd8e_4438_87b1_ade1d475dc9a.slice/crio-1938bdcbad703b1091ad453e79783c8338b27e77ea035b15a7563c240afd2c35 WatchSource:0}: Error finding container 1938bdcbad703b1091ad453e79783c8338b27e77ea035b15a7563c240afd2c35: Status 404 returned error can't find the container with id 1938bdcbad703b1091ad453e79783c8338b27e77ea035b15a7563c240afd2c35 Nov 03 00:00:01.227315 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.227279 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25"] Nov 03 00:00:01.232061 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:00:01.232027 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513fd003_68d9_4e05_bf59_3c3369882a4b.slice/crio-d22cdfff513030afc6441ef9f4553b29af7481b19446ef3bbd1a9859396d1c74 WatchSource:0}: Error finding container d22cdfff513030afc6441ef9f4553b29af7481b19446ef3bbd1a9859396d1c74: Status 404 returned error can't find the container with id d22cdfff513030afc6441ef9f4553b29af7481b19446ef3bbd1a9859396d1c74 Nov 03 00:00:01.280359 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.279287 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm"] Nov 03 00:00:01.282765 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:00:01.282727 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7131cd26_6c33_4d0f_a629_d390bc70b0c2.slice/crio-0244df101ab071b0df5edd1cf5d7e4f384a5a3dd5c80e2c582412200f8d6fdc8 WatchSource:0}: Error finding container 0244df101ab071b0df5edd1cf5d7e4f384a5a3dd5c80e2c582412200f8d6fdc8: Status 404 returned error can't find the container with id 0244df101ab071b0df5edd1cf5d7e4f384a5a3dd5c80e2c582412200f8d6fdc8 Nov 03 00:00:01.317480 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.317454 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg"] Nov 03 00:00:01.323486 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.322466 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf"] Nov 03 00:00:01.334311 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.334226 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj"] Nov 03 00:00:01.368629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.368597 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b"] Nov 03 00:00:01.417897 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.417855 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm"] Nov 03 00:00:01.489507 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.489476 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4"] Nov 03 00:00:01.497247 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:00:01.497220 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f4c6ca_d293_4bfe_b736_888f35359964.slice/crio-5fc68c60d86540839fe23394bd37644f267e283ac826278b53751bfc511cbf45 WatchSource:0}: Error finding container 5fc68c60d86540839fe23394bd37644f267e283ac826278b53751bfc511cbf45: Status 404 returned error can't find the container with id 5fc68c60d86540839fe23394bd37644f267e283ac826278b53751bfc511cbf45 Nov 03 00:00:01.511180 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:00:01.510503 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf3fac2_f23d_49a6_b27a_b395e1722d75.slice/crio-d29b74450a8f6d562c0df11aab61d31bb23e73305630a7f5e6a688779e9c03aa WatchSource:0}: Error finding container d29b74450a8f6d562c0df11aab61d31bb23e73305630a7f5e6a688779e9c03aa: Status 404 returned error can't find the container with id d29b74450a8f6d562c0df11aab61d31bb23e73305630a7f5e6a688779e9c03aa Nov 03 00:00:01.511576 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.511478 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz"] Nov 03 00:00:01.896418 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.896133 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" event={"ID":"8c2c9c15-097d-4fc2-a471-deb4c54dad7c","Type":"ContainerStarted","Data":"ef0290d6ef9bc57e3bcbc95d3f395d268bc6ccdc236a327842b5cf114321aa57"} Nov 03 00:00:01.896418 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.896222 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" event={"ID":"8c2c9c15-097d-4fc2-a471-deb4c54dad7c","Type":"ContainerStarted","Data":"0ae587d8e97a8ba399323265704a1fb8d62b1f1373e5635aca63037ab56c90c3"} Nov 03 00:00:01.898420 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.898114 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" event={"ID":"c6b2d887-1c8e-4028-98be-5c90ee7bb36d","Type":"ContainerStarted","Data":"f03f5075c02fc49fb7c6c5e9645ed02cc652d96c4d99427363c48ecb9cd04a77"} Nov 03 00:00:01.898420 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.898194 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" event={"ID":"c6b2d887-1c8e-4028-98be-5c90ee7bb36d","Type":"ContainerStarted","Data":"74dac4a8eb1f3e89c6074af4a59ea802e570b9f2f100f4dd3a84619f756e1159"} Nov 03 00:00:01.899799 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.899776 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" event={"ID":"85f4c6ca-d293-4bfe-b736-888f35359964","Type":"ContainerStarted","Data":"5fc68c60d86540839fe23394bd37644f267e283ac826278b53751bfc511cbf45"} Nov 03 00:00:01.901648 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.901622 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" event={"ID":"5610bb86-f95e-4ee3-81db-63edc417a640","Type":"ContainerStarted","Data":"4ec58d3a2590c4b3f37d8f6b51deccb2a30934fd92917e0d3ed7f991282b972a"} Nov 03 00:00:01.901761 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.901669 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" event={"ID":"5610bb86-f95e-4ee3-81db-63edc417a640","Type":"ContainerStarted","Data":"4101f0daa079ae31457b19ed49dd4ba2975672b3c958f6cf1c3e61827ec327b7"} Nov 03 00:00:01.904332 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.904311 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" event={"ID":"d5dee99d-d0f4-4e91-abdf-b21758d550d4","Type":"ContainerStarted","Data":"46cff7f76c0572550b6f31906c5654f89693077af5c5e9594927d9d564ed7ab6"} Nov 03 00:00:01.905311 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.905240 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" event={"ID":"d5dee99d-d0f4-4e91-abdf-b21758d550d4","Type":"ContainerStarted","Data":"3ec726ac20212c23bf312ec4ed12d637898666e37cddfff67bfc70194447b9fa"} Nov 03 00:00:01.906516 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.906498 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" event={"ID":"79a2752d-fd8e-4438-87b1-ade1d475dc9a","Type":"ContainerStarted","Data":"221ec02f38db912285f89f8fcb852924bfee262362d5556d6e23bb99daf4af82"} Nov 03 00:00:01.907018 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.907004 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" event={"ID":"79a2752d-fd8e-4438-87b1-ade1d475dc9a","Type":"ContainerStarted","Data":"1938bdcbad703b1091ad453e79783c8338b27e77ea035b15a7563c240afd2c35"} Nov 03 00:00:01.908282 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.908265 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" event={"ID":"295e9734-7b84-4a80-a5d7-11d68964f215","Type":"ContainerStarted","Data":"5b55bac55c5314c13f785a7c7dce4ac93f7f22a4885011278d659b9e1cfd1d28"} Nov 03 00:00:01.910073 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.909712 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" event={"ID":"7131cd26-6c33-4d0f-a629-d390bc70b0c2","Type":"ContainerStarted","Data":"ce2f6eafbdb15426f05f7c89d03e4a7a90ca2863957f468e09d28105d3e787b4"} Nov 03 00:00:01.910073 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.909745 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" event={"ID":"7131cd26-6c33-4d0f-a629-d390bc70b0c2","Type":"ContainerStarted","Data":"0244df101ab071b0df5edd1cf5d7e4f384a5a3dd5c80e2c582412200f8d6fdc8"} Nov 03 00:00:01.911215 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.911136 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" event={"ID":"2cf3fac2-f23d-49a6-b27a-b395e1722d75","Type":"ContainerStarted","Data":"d29b74450a8f6d562c0df11aab61d31bb23e73305630a7f5e6a688779e9c03aa"} Nov 03 00:00:01.916171 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.913242 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" event={"ID":"513fd003-68d9-4e05-bf59-3c3369882a4b","Type":"ContainerStarted","Data":"f3ec6eebbfc014ecd0c0a0894ec9319f56374f902a8e93e1c64974012325166b"} Nov 03 00:00:01.916171 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:01.913276 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" event={"ID":"513fd003-68d9-4e05-bf59-3c3369882a4b","Type":"ContainerStarted","Data":"d22cdfff513030afc6441ef9f4553b29af7481b19446ef3bbd1a9859396d1c74"} Nov 03 00:00:02.011245 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.011115 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" podStartSLOduration=2.011097756 podStartE2EDuration="2.011097756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:02.007954246 +0000 UTC m=+3995.415266075" watchObservedRunningTime="2026-11-03 00:00:02.011097756 +0000 UTC m=+3995.418409587" Nov 03 00:00:02.236965 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.234951 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" podStartSLOduration=2.23493291 podStartE2EDuration="2.23493291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:02.233553489 +0000 UTC m=+3995.640865319" watchObservedRunningTime="2026-11-03 00:00:02.23493291 +0000 UTC m=+3995.642244741" Nov 03 00:00:02.236965 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.236679 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" podStartSLOduration=2.236668072 podStartE2EDuration="2.236668072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:02.121551764 +0000 UTC m=+3995.528863594" watchObservedRunningTime="2026-11-03 00:00:02.236668072 +0000 UTC m=+3995.643979903" Nov 03 00:00:02.349056 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.346934 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" podStartSLOduration=2.34691543 podStartE2EDuration="2.34691543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:02.344517796 +0000 UTC m=+3995.751829625" watchObservedRunningTime="2026-11-03 00:00:02.34691543 +0000 UTC m=+3995.754227261" Nov 03 00:00:02.565262 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.553986 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" podStartSLOduration=2.553966023 podStartE2EDuration="2.553966023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:02.495766716 +0000 UTC m=+3995.903078549" watchObservedRunningTime="2026-11-03 00:00:02.553966023 +0000 UTC m=+3995.961277853" Nov 03 00:00:02.614595 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.614528 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" podStartSLOduration=2.614507523 podStartE2EDuration="2.614507523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:02.604031683 +0000 UTC m=+3996.011343513" watchObservedRunningTime="2026-11-03 00:00:02.614507523 +0000 UTC m=+3996.021819352" Nov 03 00:00:02.716211 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.716161 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" podStartSLOduration=2.716135639 podStartE2EDuration="2.716135639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:02.712852009 +0000 UTC m=+3996.120163839" watchObservedRunningTime="2026-11-03 00:00:02.716135639 +0000 UTC m=+3996.123447467" Nov 03 00:00:02.922204 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.922174 5672 generic.go:356] "Generic (PLEG): container finished" podID="295e9734-7b84-4a80-a5d7-11d68964f215" containerID="832816655f529026df411fb98e1335d963fb1199f6ae364604291dee82293641" exitCode=60 Nov 03 00:00:02.922415 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.922257 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" event={"ID":"295e9734-7b84-4a80-a5d7-11d68964f215","Type":"ContainerDied","Data":"832816655f529026df411fb98e1335d963fb1199f6ae364604291dee82293641"} Nov 03 00:00:02.922546 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.922526 5672 scope.go:122] "RemoveContainer" containerID="832816655f529026df411fb98e1335d963fb1199f6ae364604291dee82293641" Nov 03 00:00:02.924202 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.924144 5672 generic.go:356] "Generic (PLEG): container finished" podID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerID="ce2f6eafbdb15426f05f7c89d03e4a7a90ca2863957f468e09d28105d3e787b4" exitCode=60 Nov 03 00:00:02.924349 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.924263 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" event={"ID":"7131cd26-6c33-4d0f-a629-d390bc70b0c2","Type":"ContainerDied","Data":"ce2f6eafbdb15426f05f7c89d03e4a7a90ca2863957f468e09d28105d3e787b4"} Nov 03 00:00:02.924496 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.924474 5672 scope.go:122] "RemoveContainer" containerID="ce2f6eafbdb15426f05f7c89d03e4a7a90ca2863957f468e09d28105d3e787b4" Nov 03 00:00:02.935857 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.935829 5672 generic.go:356] "Generic (PLEG): container finished" podID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerID="f0dd18ac3190957541f37c8d8ee130df343516b479a083c8834ac664a4987cd4" exitCode=60 Nov 03 00:00:02.936239 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.936221 5672 scope.go:122] "RemoveContainer" containerID="f0dd18ac3190957541f37c8d8ee130df343516b479a083c8834ac664a4987cd4" Nov 03 00:00:02.936574 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.936548 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" event={"ID":"2cf3fac2-f23d-49a6-b27a-b395e1722d75","Type":"ContainerDied","Data":"f0dd18ac3190957541f37c8d8ee130df343516b479a083c8834ac664a4987cd4"} Nov 03 00:00:02.938860 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.938667 5672 generic.go:356] "Generic (PLEG): container finished" podID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerID="f3ec6eebbfc014ecd0c0a0894ec9319f56374f902a8e93e1c64974012325166b" exitCode=60 Nov 03 00:00:02.938860 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.938805 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" event={"ID":"513fd003-68d9-4e05-bf59-3c3369882a4b","Type":"ContainerDied","Data":"f3ec6eebbfc014ecd0c0a0894ec9319f56374f902a8e93e1c64974012325166b"} Nov 03 00:00:02.939017 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.938935 5672 scope.go:122] "RemoveContainer" containerID="f3ec6eebbfc014ecd0c0a0894ec9319f56374f902a8e93e1c64974012325166b" Nov 03 00:00:02.941312 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.941238 5672 generic.go:356] "Generic (PLEG): container finished" podID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerID="ef0290d6ef9bc57e3bcbc95d3f395d268bc6ccdc236a327842b5cf114321aa57" exitCode=60 Nov 03 00:00:02.942074 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.941486 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" event={"ID":"8c2c9c15-097d-4fc2-a471-deb4c54dad7c","Type":"ContainerDied","Data":"ef0290d6ef9bc57e3bcbc95d3f395d268bc6ccdc236a327842b5cf114321aa57"} Nov 03 00:00:02.942074 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.941778 5672 scope.go:122] "RemoveContainer" containerID="ef0290d6ef9bc57e3bcbc95d3f395d268bc6ccdc236a327842b5cf114321aa57" Nov 03 00:00:02.944313 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.944289 5672 generic.go:356] "Generic (PLEG): container finished" podID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerID="f03f5075c02fc49fb7c6c5e9645ed02cc652d96c4d99427363c48ecb9cd04a77" exitCode=60 Nov 03 00:00:02.944627 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.944558 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" event={"ID":"c6b2d887-1c8e-4028-98be-5c90ee7bb36d","Type":"ContainerDied","Data":"f03f5075c02fc49fb7c6c5e9645ed02cc652d96c4d99427363c48ecb9cd04a77"} Nov 03 00:00:02.944740 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.944627 5672 scope.go:122] "RemoveContainer" containerID="f03f5075c02fc49fb7c6c5e9645ed02cc652d96c4d99427363c48ecb9cd04a77" Nov 03 00:00:02.946902 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.946877 5672 generic.go:356] "Generic (PLEG): container finished" podID="85f4c6ca-d293-4bfe-b736-888f35359964" containerID="67dd5ce3f059b0be8474f8175db63195dda30af6cb40333a6e8ca05c51dc780f" exitCode=60 Nov 03 00:00:02.947192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.947126 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" event={"ID":"85f4c6ca-d293-4bfe-b736-888f35359964","Type":"ContainerDied","Data":"67dd5ce3f059b0be8474f8175db63195dda30af6cb40333a6e8ca05c51dc780f"} Nov 03 00:00:02.947304 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.947254 5672 scope.go:122] "RemoveContainer" containerID="67dd5ce3f059b0be8474f8175db63195dda30af6cb40333a6e8ca05c51dc780f" Nov 03 00:00:02.950198 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.950174 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" event={"ID":"5610bb86-f95e-4ee3-81db-63edc417a640","Type":"ContainerDied","Data":"4ec58d3a2590c4b3f37d8f6b51deccb2a30934fd92917e0d3ed7f991282b972a"} Nov 03 00:00:02.950451 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.950430 5672 scope.go:122] "RemoveContainer" containerID="4ec58d3a2590c4b3f37d8f6b51deccb2a30934fd92917e0d3ed7f991282b972a" Nov 03 00:00:02.950841 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.950028 5672 generic.go:356] "Generic (PLEG): container finished" podID="5610bb86-f95e-4ee3-81db-63edc417a640" containerID="4ec58d3a2590c4b3f37d8f6b51deccb2a30934fd92917e0d3ed7f991282b972a" exitCode=60 Nov 03 00:00:02.959602 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.959449 5672 generic.go:356] "Generic (PLEG): container finished" podID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerID="46cff7f76c0572550b6f31906c5654f89693077af5c5e9594927d9d564ed7ab6" exitCode=60 Nov 03 00:00:02.959602 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.959536 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" event={"ID":"d5dee99d-d0f4-4e91-abdf-b21758d550d4","Type":"ContainerDied","Data":"46cff7f76c0572550b6f31906c5654f89693077af5c5e9594927d9d564ed7ab6"} Nov 03 00:00:02.962949 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.959803 5672 scope.go:122] "RemoveContainer" containerID="46cff7f76c0572550b6f31906c5654f89693077af5c5e9594927d9d564ed7ab6" Nov 03 00:00:02.966231 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.966210 5672 generic.go:356] "Generic (PLEG): container finished" podID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerID="221ec02f38db912285f89f8fcb852924bfee262362d5556d6e23bb99daf4af82" exitCode=60 Nov 03 00:00:02.967765 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.966647 5672 scope.go:122] "RemoveContainer" containerID="221ec02f38db912285f89f8fcb852924bfee262362d5556d6e23bb99daf4af82" Nov 03 00:00:02.967894 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:02.966661 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" event={"ID":"79a2752d-fd8e-4438-87b1-ade1d475dc9a","Type":"ContainerDied","Data":"221ec02f38db912285f89f8fcb852924bfee262362d5556d6e23bb99daf4af82"} Nov 03 00:00:03.976400 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:03.976367 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" event={"ID":"7131cd26-6c33-4d0f-a629-d390bc70b0c2","Type":"ContainerStarted","Data":"3fd4e9869e87789fb1f9df4341bab48bf84343f3aa990decf0e0f9f9c4ba5834"} Nov 03 00:00:03.978453 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:03.978422 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" event={"ID":"2cf3fac2-f23d-49a6-b27a-b395e1722d75","Type":"ContainerStarted","Data":"324298c0d5fa3f300504597d436c8e0e4ea7ae0f95db2a50baf51c0b9dc0db3b"} Nov 03 00:00:03.980385 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:03.980355 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" event={"ID":"513fd003-68d9-4e05-bf59-3c3369882a4b","Type":"ContainerStarted","Data":"c36dbd8e1398f7316c78b0710c1f7f20b12454bb9f1596a8eef3244514b2082e"} Nov 03 00:00:03.982318 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:03.982292 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" event={"ID":"8c2c9c15-097d-4fc2-a471-deb4c54dad7c","Type":"ContainerStarted","Data":"523dad71bb2da36635b6887399deac20ea1252064a70ecf5d6c88b79fab3bf68"} Nov 03 00:00:03.984331 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:03.984306 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" event={"ID":"c6b2d887-1c8e-4028-98be-5c90ee7bb36d","Type":"ContainerStarted","Data":"9996bb00408cf426af2509d814f0f16c16b2e99745a2daaf285ab6ff59b9ed5f"} Nov 03 00:00:03.986507 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:03.986482 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" event={"ID":"85f4c6ca-d293-4bfe-b736-888f35359964","Type":"ContainerStarted","Data":"96d029df3d202837737835a6683117bc7797f5d9e266c65707666602ae27887f"} Nov 03 00:00:03.988684 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:03.988635 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" event={"ID":"5610bb86-f95e-4ee3-81db-63edc417a640","Type":"ContainerStarted","Data":"c8f54e6747178becbd82d64f08abd5ac829112d5d6987cce517b634e827fbd21"} Nov 03 00:00:03.990659 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:03.990636 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" event={"ID":"d5dee99d-d0f4-4e91-abdf-b21758d550d4","Type":"ContainerStarted","Data":"5259e449e01c3e19f8c155c27b59c4635b1ce5639938d68dc624a0d3b591b2c1"} Nov 03 00:00:03.992612 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:03.992585 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" event={"ID":"79a2752d-fd8e-4438-87b1-ade1d475dc9a","Type":"ContainerStarted","Data":"15e5b3a3408c4ddb6cf10c4edf48772291dfd7b0a1e51009ac5fe8548488eead"} Nov 03 00:00:03.994426 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:03.994399 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" event={"ID":"295e9734-7b84-4a80-a5d7-11d68964f215","Type":"ContainerStarted","Data":"e6319d8a64c19cd5cf36bff81c342688e5d139379652257422aab51c0b4d7873"} Nov 03 00:00:04.219808 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:04.219744 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" podStartSLOduration=4.219728138 podStartE2EDuration="4.219728138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:04.217448005 +0000 UTC m=+3997.624759836" watchObservedRunningTime="2026-11-03 00:00:04.219728138 +0000 UTC m=+3997.627039967" Nov 03 00:00:04.749763 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:04.749711 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" podStartSLOduration=4.749699394 podStartE2EDuration="4.749699394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:04.747875445 +0000 UTC m=+3998.155187275" watchObservedRunningTime="2026-11-03 00:00:04.749699394 +0000 UTC m=+3998.157011222" Nov 03 00:00:04.857879 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:04.857783 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" podStartSLOduration=4.857769329 podStartE2EDuration="4.857769329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:04.854918012 +0000 UTC m=+3998.262229845" watchObservedRunningTime="2026-11-03 00:00:04.857769329 +0000 UTC m=+3998.265081158" Nov 03 00:00:05.004720 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.004685 5672 generic.go:356] "Generic (PLEG): container finished" podID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerID="c36dbd8e1398f7316c78b0710c1f7f20b12454bb9f1596a8eef3244514b2082e" exitCode=60 Nov 03 00:00:05.005220 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.004771 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" event={"ID":"513fd003-68d9-4e05-bf59-3c3369882a4b","Type":"ContainerDied","Data":"c36dbd8e1398f7316c78b0710c1f7f20b12454bb9f1596a8eef3244514b2082e"} Nov 03 00:00:05.005220 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.004827 5672 scope.go:122] "RemoveContainer" containerID="f3ec6eebbfc014ecd0c0a0894ec9319f56374f902a8e93e1c64974012325166b" Nov 03 00:00:05.005220 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.005048 5672 scope.go:122] "RemoveContainer" containerID="c36dbd8e1398f7316c78b0710c1f7f20b12454bb9f1596a8eef3244514b2082e" Nov 03 00:00:05.005406 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:05.005387 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25_odh-ai-gateway-infra(513fd003-68d9-4e05-bf59-3c3369882a4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" Nov 03 00:00:05.007003 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.006980 5672 generic.go:356] "Generic (PLEG): container finished" podID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerID="523dad71bb2da36635b6887399deac20ea1252064a70ecf5d6c88b79fab3bf68" exitCode=60 Nov 03 00:00:05.007106 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.007019 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" event={"ID":"8c2c9c15-097d-4fc2-a471-deb4c54dad7c","Type":"ContainerDied","Data":"523dad71bb2da36635b6887399deac20ea1252064a70ecf5d6c88b79fab3bf68"} Nov 03 00:00:05.007422 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.007403 5672 scope.go:122] "RemoveContainer" containerID="523dad71bb2da36635b6887399deac20ea1252064a70ecf5d6c88b79fab3bf68" Nov 03 00:00:05.007854 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:05.007831 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg_odh-ai-gateway-infra(8c2c9c15-097d-4fc2-a471-deb4c54dad7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" Nov 03 00:00:05.009347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.009325 5672 generic.go:356] "Generic (PLEG): container finished" podID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerID="9996bb00408cf426af2509d814f0f16c16b2e99745a2daaf285ab6ff59b9ed5f" exitCode=60 Nov 03 00:00:05.009454 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.009393 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" event={"ID":"c6b2d887-1c8e-4028-98be-5c90ee7bb36d","Type":"ContainerDied","Data":"9996bb00408cf426af2509d814f0f16c16b2e99745a2daaf285ab6ff59b9ed5f"} Nov 03 00:00:05.009654 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.009639 5672 scope.go:122] "RemoveContainer" containerID="9996bb00408cf426af2509d814f0f16c16b2e99745a2daaf285ab6ff59b9ed5f" Nov 03 00:00:05.009896 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:05.009875 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b_odh-ai-gateway-infra(c6b2d887-1c8e-4028-98be-5c90ee7bb36d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" Nov 03 00:00:05.011499 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.011479 5672 generic.go:356] "Generic (PLEG): container finished" podID="85f4c6ca-d293-4bfe-b736-888f35359964" containerID="96d029df3d202837737835a6683117bc7797f5d9e266c65707666602ae27887f" exitCode=60 Nov 03 00:00:05.011615 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.011506 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" event={"ID":"85f4c6ca-d293-4bfe-b736-888f35359964","Type":"ContainerDied","Data":"96d029df3d202837737835a6683117bc7797f5d9e266c65707666602ae27887f"} Nov 03 00:00:05.011802 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.011786 5672 scope.go:122] "RemoveContainer" containerID="96d029df3d202837737835a6683117bc7797f5d9e266c65707666602ae27887f" Nov 03 00:00:05.012027 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:05.012006 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4_odh-ai-gateway-infra(85f4c6ca-d293-4bfe-b736-888f35359964)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" Nov 03 00:00:05.013789 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.013765 5672 generic.go:356] "Generic (PLEG): container finished" podID="5610bb86-f95e-4ee3-81db-63edc417a640" containerID="c8f54e6747178becbd82d64f08abd5ac829112d5d6987cce517b634e827fbd21" exitCode=60 Nov 03 00:00:05.013877 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.013844 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" event={"ID":"5610bb86-f95e-4ee3-81db-63edc417a640","Type":"ContainerDied","Data":"c8f54e6747178becbd82d64f08abd5ac829112d5d6987cce517b634e827fbd21"} Nov 03 00:00:05.014207 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.014186 5672 scope.go:122] "RemoveContainer" containerID="c8f54e6747178becbd82d64f08abd5ac829112d5d6987cce517b634e827fbd21" Nov 03 00:00:05.014502 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:05.014479 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj_odh-ai-gateway-infra(5610bb86-f95e-4ee3-81db-63edc417a640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" Nov 03 00:00:05.015848 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.015818 5672 generic.go:356] "Generic (PLEG): container finished" podID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerID="5259e449e01c3e19f8c155c27b59c4635b1ce5639938d68dc624a0d3b591b2c1" exitCode=60 Nov 03 00:00:05.015932 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.015896 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" event={"ID":"d5dee99d-d0f4-4e91-abdf-b21758d550d4","Type":"ContainerDied","Data":"5259e449e01c3e19f8c155c27b59c4635b1ce5639938d68dc624a0d3b591b2c1"} Nov 03 00:00:05.016089 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.016073 5672 scope.go:122] "RemoveContainer" containerID="5259e449e01c3e19f8c155c27b59c4635b1ce5639938d68dc624a0d3b591b2c1" Nov 03 00:00:05.016349 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:05.016326 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf_odh-ai-gateway-infra(d5dee99d-d0f4-4e91-abdf-b21758d550d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" Nov 03 00:00:05.017888 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.017866 5672 generic.go:356] "Generic (PLEG): container finished" podID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerID="15e5b3a3408c4ddb6cf10c4edf48772291dfd7b0a1e51009ac5fe8548488eead" exitCode=60 Nov 03 00:00:05.017967 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.017906 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" event={"ID":"79a2752d-fd8e-4438-87b1-ade1d475dc9a","Type":"ContainerDied","Data":"15e5b3a3408c4ddb6cf10c4edf48772291dfd7b0a1e51009ac5fe8548488eead"} Nov 03 00:00:05.018161 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.018136 5672 scope.go:122] "RemoveContainer" containerID="15e5b3a3408c4ddb6cf10c4edf48772291dfd7b0a1e51009ac5fe8548488eead" Nov 03 00:00:05.018439 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:05.018415 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b_odh-ai-gateway-infra(79a2752d-fd8e-4438-87b1-ade1d475dc9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" Nov 03 00:00:05.024478 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.024457 5672 scope.go:122] "RemoveContainer" containerID="ef0290d6ef9bc57e3bcbc95d3f395d268bc6ccdc236a327842b5cf114321aa57" Nov 03 00:00:05.025564 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.025537 5672 generic.go:356] "Generic (PLEG): container finished" podID="295e9734-7b84-4a80-a5d7-11d68964f215" containerID="e6319d8a64c19cd5cf36bff81c342688e5d139379652257422aab51c0b4d7873" exitCode=60 Nov 03 00:00:05.025632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.025613 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" event={"ID":"295e9734-7b84-4a80-a5d7-11d68964f215","Type":"ContainerDied","Data":"e6319d8a64c19cd5cf36bff81c342688e5d139379652257422aab51c0b4d7873"} Nov 03 00:00:05.025816 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.025799 5672 scope.go:122] "RemoveContainer" containerID="e6319d8a64c19cd5cf36bff81c342688e5d139379652257422aab51c0b4d7873" Nov 03 00:00:05.026084 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:05.026061 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm_odh-ai-gateway-infra(295e9734-7b84-4a80-a5d7-11d68964f215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" Nov 03 00:00:05.027724 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.027706 5672 generic.go:356] "Generic (PLEG): container finished" podID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerID="3fd4e9869e87789fb1f9df4341bab48bf84343f3aa990decf0e0f9f9c4ba5834" exitCode=60 Nov 03 00:00:05.027828 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.027771 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" event={"ID":"7131cd26-6c33-4d0f-a629-d390bc70b0c2","Type":"ContainerDied","Data":"3fd4e9869e87789fb1f9df4341bab48bf84343f3aa990decf0e0f9f9c4ba5834"} Nov 03 00:00:05.027967 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.027951 5672 scope.go:122] "RemoveContainer" containerID="3fd4e9869e87789fb1f9df4341bab48bf84343f3aa990decf0e0f9f9c4ba5834" Nov 03 00:00:05.028204 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:05.028181 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm_odh-ai-gateway-infra(7131cd26-6c33-4d0f-a629-d390bc70b0c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" Nov 03 00:00:05.029556 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.029537 5672 generic.go:356] "Generic (PLEG): container finished" podID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerID="324298c0d5fa3f300504597d436c8e0e4ea7ae0f95db2a50baf51c0b9dc0db3b" exitCode=60 Nov 03 00:00:05.029653 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.029581 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" event={"ID":"2cf3fac2-f23d-49a6-b27a-b395e1722d75","Type":"ContainerDied","Data":"324298c0d5fa3f300504597d436c8e0e4ea7ae0f95db2a50baf51c0b9dc0db3b"} Nov 03 00:00:05.029803 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.029788 5672 scope.go:122] "RemoveContainer" containerID="324298c0d5fa3f300504597d436c8e0e4ea7ae0f95db2a50baf51c0b9dc0db3b" Nov 03 00:00:05.029992 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:05.029973 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz_odh-ai-gateway-infra(2cf3fac2-f23d-49a6-b27a-b395e1722d75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" Nov 03 00:00:05.038296 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.038280 5672 scope.go:122] "RemoveContainer" containerID="f03f5075c02fc49fb7c6c5e9645ed02cc652d96c4d99427363c48ecb9cd04a77" Nov 03 00:00:05.050474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.050452 5672 scope.go:122] "RemoveContainer" containerID="67dd5ce3f059b0be8474f8175db63195dda30af6cb40333a6e8ca05c51dc780f" Nov 03 00:00:05.065534 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.065515 5672 scope.go:122] "RemoveContainer" containerID="4ec58d3a2590c4b3f37d8f6b51deccb2a30934fd92917e0d3ed7f991282b972a" Nov 03 00:00:05.080636 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.080611 5672 scope.go:122] "RemoveContainer" containerID="46cff7f76c0572550b6f31906c5654f89693077af5c5e9594927d9d564ed7ab6" Nov 03 00:00:05.093493 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.093473 5672 scope.go:122] "RemoveContainer" containerID="221ec02f38db912285f89f8fcb852924bfee262362d5556d6e23bb99daf4af82" Nov 03 00:00:05.105922 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.105898 5672 scope.go:122] "RemoveContainer" containerID="832816655f529026df411fb98e1335d963fb1199f6ae364604291dee82293641" Nov 03 00:00:05.118241 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.118223 5672 scope.go:122] "RemoveContainer" containerID="ce2f6eafbdb15426f05f7c89d03e4a7a90ca2863957f468e09d28105d3e787b4" Nov 03 00:00:05.130250 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:05.130219 5672 scope.go:122] "RemoveContainer" containerID="f0dd18ac3190957541f37c8d8ee130df343516b479a083c8834ac664a4987cd4" Nov 03 00:00:06.038593 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:06.038560 5672 scope.go:122] "RemoveContainer" containerID="e6319d8a64c19cd5cf36bff81c342688e5d139379652257422aab51c0b4d7873" Nov 03 00:00:06.039054 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:06.038865 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm_odh-ai-gateway-infra(295e9734-7b84-4a80-a5d7-11d68964f215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" Nov 03 00:00:06.040231 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:06.040209 5672 scope.go:122] "RemoveContainer" containerID="3fd4e9869e87789fb1f9df4341bab48bf84343f3aa990decf0e0f9f9c4ba5834" Nov 03 00:00:06.040514 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:06.040486 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm_odh-ai-gateway-infra(7131cd26-6c33-4d0f-a629-d390bc70b0c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" Nov 03 00:00:06.041743 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:06.041721 5672 scope.go:122] "RemoveContainer" containerID="324298c0d5fa3f300504597d436c8e0e4ea7ae0f95db2a50baf51c0b9dc0db3b" Nov 03 00:00:06.041999 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:06.041974 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz_odh-ai-gateway-infra(2cf3fac2-f23d-49a6-b27a-b395e1722d75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" Nov 03 00:00:06.043259 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:06.043241 5672 scope.go:122] "RemoveContainer" containerID="c36dbd8e1398f7316c78b0710c1f7f20b12454bb9f1596a8eef3244514b2082e" Nov 03 00:00:06.043512 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:06.043487 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25_odh-ai-gateway-infra(513fd003-68d9-4e05-bf59-3c3369882a4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" Nov 03 00:00:06.044789 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:06.044773 5672 scope.go:122] "RemoveContainer" containerID="523dad71bb2da36635b6887399deac20ea1252064a70ecf5d6c88b79fab3bf68" Nov 03 00:00:06.045037 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:06.045010 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg_odh-ai-gateway-infra(8c2c9c15-097d-4fc2-a471-deb4c54dad7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" Nov 03 00:00:06.046362 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:06.046344 5672 scope.go:122] "RemoveContainer" containerID="9996bb00408cf426af2509d814f0f16c16b2e99745a2daaf285ab6ff59b9ed5f" Nov 03 00:00:06.046598 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:06.046574 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b_odh-ai-gateway-infra(c6b2d887-1c8e-4028-98be-5c90ee7bb36d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" Nov 03 00:00:06.047829 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:06.047812 5672 scope.go:122] "RemoveContainer" containerID="96d029df3d202837737835a6683117bc7797f5d9e266c65707666602ae27887f" Nov 03 00:00:06.048002 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:06.047984 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4_odh-ai-gateway-infra(85f4c6ca-d293-4bfe-b736-888f35359964)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" Nov 03 00:00:06.049344 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:06.049327 5672 scope.go:122] "RemoveContainer" containerID="c8f54e6747178becbd82d64f08abd5ac829112d5d6987cce517b634e827fbd21" Nov 03 00:00:06.049507 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:06.049491 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj_odh-ai-gateway-infra(5610bb86-f95e-4ee3-81db-63edc417a640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" Nov 03 00:00:06.050866 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:06.050851 5672 scope.go:122] "RemoveContainer" containerID="5259e449e01c3e19f8c155c27b59c4635b1ce5639938d68dc624a0d3b591b2c1" Nov 03 00:00:06.051041 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:06.051024 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf_odh-ai-gateway-infra(d5dee99d-d0f4-4e91-abdf-b21758d550d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" Nov 03 00:00:06.052488 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:06.052472 5672 scope.go:122] "RemoveContainer" containerID="15e5b3a3408c4ddb6cf10c4edf48772291dfd7b0a1e51009ac5fe8548488eead" Nov 03 00:00:06.052632 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:06.052618 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b_odh-ai-gateway-infra(79a2752d-fd8e-4438-87b1-ade1d475dc9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" Nov 03 00:00:09.314903 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.314869 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42"] Nov 03 00:00:09.320432 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.320406 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.409519 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.409487 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.409686 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.409524 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79qj\" (UniqueName: \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-kube-api-access-d79qj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.409686 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.409553 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.409758 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.409696 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.409758 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.409720 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.409871 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.409844 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.409929 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.409913 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.409981 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.409966 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.410115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.410094 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.503410 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.503373 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42"] Nov 03 00:00:09.511839 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.511806 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.512013 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.511856 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79qj\" (UniqueName: \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-kube-api-access-d79qj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.512013 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.511888 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.512013 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.511935 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.512013 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.511952 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.512013 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.512001 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.512272 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.512033 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.512272 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.512066 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.512272 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.512184 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.512584 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.512559 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.512783 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.512767 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.513360 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.513332 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.513574 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.513532 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.513980 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.513954 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.514982 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.514956 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.515710 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.515681 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.566814 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.566733 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.567777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.567750 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79qj\" (UniqueName: \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-kube-api-access-d79qj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qcj42\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:09.638283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:09.638257 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:10.058563 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:10.058528 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42"] Nov 03 00:00:10.065345 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:10.065314 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:10.065430 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:10.065378 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:10.065430 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:10.065405 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:10.086793 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:10.086763 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" event={"ID":"a39ffff0-85c2-44c0-b5c6-2b8dbb779079","Type":"ContainerStarted","Data":"92b9ea8d65f83ca45c22cc72b4ac4f1852263f38d346f8ad88432a88463fe12f"} Nov 03 00:00:11.096236 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:11.096195 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" event={"ID":"a39ffff0-85c2-44c0-b5c6-2b8dbb779079","Type":"ContainerStarted","Data":"603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43"} Nov 03 00:00:11.310169 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:11.309322 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" podStartSLOduration=2.3093092 podStartE2EDuration="2.3093092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:11.258681854 +0000 UTC m=+4004.665993685" watchObservedRunningTime="2026-11-03 00:00:11.3093092 +0000 UTC m=+4004.716621027" Nov 03 00:00:11.643172 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:11.640527 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:11.655225 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:11.644248 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qcj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 00:00:11.655225 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:11.644309 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 00:00:12.643172 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:12.642510 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qcj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 00:00:12.643172 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:12.642570 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 00:00:13.642170 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:13.639561 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qcj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 00:00:13.642170 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:13.639639 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 00:00:14.641914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:14.641495 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qcj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 00:00:14.641914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:14.641555 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 00:00:15.643170 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:15.642529 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qcj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 00:00:15.643170 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:15.642595 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 00:00:15.705633 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:15.705598 5672 scope.go:122] "RemoveContainer" containerID="9996bb00408cf426af2509d814f0f16c16b2e99745a2daaf285ab6ff59b9ed5f" Nov 03 00:00:15.705802 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:15.705709 5672 scope.go:122] "RemoveContainer" containerID="15e5b3a3408c4ddb6cf10c4edf48772291dfd7b0a1e51009ac5fe8548488eead" Nov 03 00:00:15.705860 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:15.705798 5672 scope.go:122] "RemoveContainer" containerID="96d029df3d202837737835a6683117bc7797f5d9e266c65707666602ae27887f" Nov 03 00:00:15.705934 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:15.705865 5672 scope.go:122] "RemoveContainer" containerID="c8f54e6747178becbd82d64f08abd5ac829112d5d6987cce517b634e827fbd21" Nov 03 00:00:16.638784 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:16.638745 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qcj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 00:00:16.638909 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:16.638812 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 00:00:17.180548 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.180514 5672 generic.go:356] "Generic (PLEG): container finished" podID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerID="4bad2461dc3feeead1e4559bb77895e8bd2dc22ceb3a642c345a5886ac814663" exitCode=60 Nov 03 00:00:17.181032 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.180599 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" event={"ID":"c6b2d887-1c8e-4028-98be-5c90ee7bb36d","Type":"ContainerDied","Data":"4bad2461dc3feeead1e4559bb77895e8bd2dc22ceb3a642c345a5886ac814663"} Nov 03 00:00:17.181032 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.180678 5672 scope.go:122] "RemoveContainer" containerID="9996bb00408cf426af2509d814f0f16c16b2e99745a2daaf285ab6ff59b9ed5f" Nov 03 00:00:17.181175 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.181063 5672 scope.go:122] "RemoveContainer" containerID="4bad2461dc3feeead1e4559bb77895e8bd2dc22ceb3a642c345a5886ac814663" Nov 03 00:00:17.181433 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:17.181411 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b_odh-ai-gateway-infra(c6b2d887-1c8e-4028-98be-5c90ee7bb36d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" Nov 03 00:00:17.183259 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.183235 5672 generic.go:356] "Generic (PLEG): container finished" podID="85f4c6ca-d293-4bfe-b736-888f35359964" containerID="b3f9510708bc87792925a95f196d794041b4bb046620b2705090f129f6871b58" exitCode=60 Nov 03 00:00:17.183374 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.183277 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" event={"ID":"85f4c6ca-d293-4bfe-b736-888f35359964","Type":"ContainerDied","Data":"b3f9510708bc87792925a95f196d794041b4bb046620b2705090f129f6871b58"} Nov 03 00:00:17.183660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.183644 5672 scope.go:122] "RemoveContainer" containerID="b3f9510708bc87792925a95f196d794041b4bb046620b2705090f129f6871b58" Nov 03 00:00:17.183952 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:17.183929 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4_odh-ai-gateway-infra(85f4c6ca-d293-4bfe-b736-888f35359964)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" Nov 03 00:00:17.185273 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.185253 5672 generic.go:356] "Generic (PLEG): container finished" podID="5610bb86-f95e-4ee3-81db-63edc417a640" containerID="b22aed0460d955c8a269e118026c987a0a8014ba73ba99d3f7c2377ac80cb5be" exitCode=60 Nov 03 00:00:17.185387 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.185330 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" event={"ID":"5610bb86-f95e-4ee3-81db-63edc417a640","Type":"ContainerDied","Data":"b22aed0460d955c8a269e118026c987a0a8014ba73ba99d3f7c2377ac80cb5be"} Nov 03 00:00:17.185676 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.185656 5672 scope.go:122] "RemoveContainer" containerID="b22aed0460d955c8a269e118026c987a0a8014ba73ba99d3f7c2377ac80cb5be" Nov 03 00:00:17.186246 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:17.185928 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj_odh-ai-gateway-infra(5610bb86-f95e-4ee3-81db-63edc417a640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" Nov 03 00:00:17.187875 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.187856 5672 generic.go:356] "Generic (PLEG): container finished" podID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerID="1ab1bef7e781d12806cf10bc67711cc2527aa714ccaf63dc5b75cc513aacf133" exitCode=60 Nov 03 00:00:17.187985 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.187900 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" event={"ID":"79a2752d-fd8e-4438-87b1-ade1d475dc9a","Type":"ContainerDied","Data":"1ab1bef7e781d12806cf10bc67711cc2527aa714ccaf63dc5b75cc513aacf133"} Nov 03 00:00:17.188243 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.188226 5672 scope.go:122] "RemoveContainer" containerID="1ab1bef7e781d12806cf10bc67711cc2527aa714ccaf63dc5b75cc513aacf133" Nov 03 00:00:17.188471 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:17.188453 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b_odh-ai-gateway-infra(79a2752d-fd8e-4438-87b1-ade1d475dc9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" Nov 03 00:00:17.194684 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.194666 5672 scope.go:122] "RemoveContainer" containerID="96d029df3d202837737835a6683117bc7797f5d9e266c65707666602ae27887f" Nov 03 00:00:17.207594 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.207561 5672 scope.go:122] "RemoveContainer" containerID="c8f54e6747178becbd82d64f08abd5ac829112d5d6987cce517b634e827fbd21" Nov 03 00:00:17.222020 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.221998 5672 scope.go:122] "RemoveContainer" containerID="15e5b3a3408c4ddb6cf10c4edf48772291dfd7b0a1e51009ac5fe8548488eead" Nov 03 00:00:17.638599 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.638568 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qcj42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 00:00:17.638814 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.638615 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 00:00:17.706607 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.706577 5672 scope.go:122] "RemoveContainer" containerID="324298c0d5fa3f300504597d436c8e0e4ea7ae0f95db2a50baf51c0b9dc0db3b" Nov 03 00:00:17.706828 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.706757 5672 scope.go:122] "RemoveContainer" containerID="c36dbd8e1398f7316c78b0710c1f7f20b12454bb9f1596a8eef3244514b2082e" Nov 03 00:00:17.706898 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.706858 5672 scope.go:122] "RemoveContainer" containerID="3fd4e9869e87789fb1f9df4341bab48bf84343f3aa990decf0e0f9f9c4ba5834" Nov 03 00:00:17.707211 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.707182 5672 scope.go:122] "RemoveContainer" containerID="523dad71bb2da36635b6887399deac20ea1252064a70ecf5d6c88b79fab3bf68" Nov 03 00:00:17.707333 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.707301 5672 scope.go:122] "RemoveContainer" containerID="5259e449e01c3e19f8c155c27b59c4635b1ce5639938d68dc624a0d3b591b2c1" Nov 03 00:00:17.707401 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:17.707377 5672 scope.go:122] "RemoveContainer" containerID="e6319d8a64c19cd5cf36bff81c342688e5d139379652257422aab51c0b4d7873" Nov 03 00:00:18.200187 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.200094 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" event={"ID":"7131cd26-6c33-4d0f-a629-d390bc70b0c2","Type":"ContainerStarted","Data":"e3e00128dffb756756964131c094061e518852b8f1ab8dd54cdc6b2454d8f787"} Nov 03 00:00:18.202328 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.202303 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" event={"ID":"513fd003-68d9-4e05-bf59-3c3369882a4b","Type":"ContainerStarted","Data":"24fbbfa7b03137cd323dfdf1009fbf5697102d198788ddd253857e6c020c3fba"} Nov 03 00:00:18.204462 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.204438 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" event={"ID":"8c2c9c15-097d-4fc2-a471-deb4c54dad7c","Type":"ContainerStarted","Data":"205bff68ca9a2c2a6b1639a4396a1a4c574eb1387f2e846b28beef7a98e61f54"} Nov 03 00:00:18.297509 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.297477 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b"] Nov 03 00:00:18.434261 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.431998 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj"] Nov 03 00:00:18.482718 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.481807 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" Nov 03 00:00:18.509060 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.509028 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b"] Nov 03 00:00:18.617971 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.617942 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4"] Nov 03 00:00:18.640404 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.639436 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" Nov 03 00:00:18.641644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.640723 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-maas-api-service-ca\") pod \"c6b2d887-1c8e-4028-98be-5c90ee7bb36d\" (UID: \"c6b2d887-1c8e-4028-98be-5c90ee7bb36d\") " Nov 03 00:00:18.641644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.640768 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-kube-api-access-v6fx8\" (UniqueName: \"kubernetes.io/projected/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-kube-api-access-v6fx8\") pod \"c6b2d887-1c8e-4028-98be-5c90ee7bb36d\" (UID: \"c6b2d887-1c8e-4028-98be-5c90ee7bb36d\") " Nov 03 00:00:18.642128 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.642025 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-maas-api-service-ca" pod "c6b2d887-1c8e-4028-98be-5c90ee7bb36d" (UID: "c6b2d887-1c8e-4028-98be-5c90ee7bb36d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:18.657183 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.656557 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-kube-api-access-v6fx8" pod "c6b2d887-1c8e-4028-98be-5c90ee7bb36d" (UID: "c6b2d887-1c8e-4028-98be-5c90ee7bb36d"). InnerVolumeSpecName "kube-api-access-v6fx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:18.717332 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.717293 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" Nov 03 00:00:18.742208 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.742137 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5610bb86-f95e-4ee3-81db-63edc417a640-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5610bb86-f95e-4ee3-81db-63edc417a640-maas-api-service-ca\") pod \"5610bb86-f95e-4ee3-81db-63edc417a640\" (UID: \"5610bb86-f95e-4ee3-81db-63edc417a640\") " Nov 03 00:00:18.747248 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.742436 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5610bb86-f95e-4ee3-81db-63edc417a640-kube-api-access-ns2fl\" (UniqueName: \"kubernetes.io/projected/5610bb86-f95e-4ee3-81db-63edc417a640-kube-api-access-ns2fl\") pod \"5610bb86-f95e-4ee3-81db-63edc417a640\" (UID: \"5610bb86-f95e-4ee3-81db-63edc417a640\") " Nov 03 00:00:18.747248 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.743287 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5610bb86-f95e-4ee3-81db-63edc417a640-maas-api-service-ca" pod "5610bb86-f95e-4ee3-81db-63edc417a640" (UID: "5610bb86-f95e-4ee3-81db-63edc417a640"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:18.747248 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.743675 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.747248 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.743698 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6fx8\" (UniqueName: \"kubernetes.io/projected/c6b2d887-1c8e-4028-98be-5c90ee7bb36d-kube-api-access-v6fx8\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.747248 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.743712 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5610bb86-f95e-4ee3-81db-63edc417a640-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.760507 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.760475 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5610bb86-f95e-4ee3-81db-63edc417a640-kube-api-access-ns2fl" pod "5610bb86-f95e-4ee3-81db-63edc417a640" (UID: "5610bb86-f95e-4ee3-81db-63edc417a640"). InnerVolumeSpecName "kube-api-access-ns2fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:18.844767 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.844372 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a2752d-fd8e-4438-87b1-ade1d475dc9a-kube-api-access-cv68l\" (UniqueName: \"kubernetes.io/projected/79a2752d-fd8e-4438-87b1-ade1d475dc9a-kube-api-access-cv68l\") pod \"79a2752d-fd8e-4438-87b1-ade1d475dc9a\" (UID: \"79a2752d-fd8e-4438-87b1-ade1d475dc9a\") " Nov 03 00:00:18.844767 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.844468 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a2752d-fd8e-4438-87b1-ade1d475dc9a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a2752d-fd8e-4438-87b1-ade1d475dc9a-maas-api-service-ca\") pod \"79a2752d-fd8e-4438-87b1-ade1d475dc9a\" (UID: \"79a2752d-fd8e-4438-87b1-ade1d475dc9a\") " Nov 03 00:00:18.845031 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.845003 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a2752d-fd8e-4438-87b1-ade1d475dc9a-maas-api-service-ca" pod "79a2752d-fd8e-4438-87b1-ade1d475dc9a" (UID: "79a2752d-fd8e-4438-87b1-ade1d475dc9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:18.845206 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.845125 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2fl\" (UniqueName: \"kubernetes.io/projected/5610bb86-f95e-4ee3-81db-63edc417a640-kube-api-access-ns2fl\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.845206 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.845170 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a2752d-fd8e-4438-87b1-ade1d475dc9a-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:18.846735 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.846711 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a2752d-fd8e-4438-87b1-ade1d475dc9a-kube-api-access-cv68l" pod "79a2752d-fd8e-4438-87b1-ade1d475dc9a" (UID: "79a2752d-fd8e-4438-87b1-ade1d475dc9a"). InnerVolumeSpecName "kube-api-access-cv68l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:18.998222 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:18.996839 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv68l\" (UniqueName: \"kubernetes.io/projected/79a2752d-fd8e-4438-87b1-ade1d475dc9a-kube-api-access-cv68l\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.113882 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.113835 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:19.114465 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.114433 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:19.115027 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.115007 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:00:19.138202 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.138178 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" Nov 03 00:00:19.198810 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.198780 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f4c6ca-d293-4bfe-b736-888f35359964-kube-api-access-zkcpj\" (UniqueName: \"kubernetes.io/projected/85f4c6ca-d293-4bfe-b736-888f35359964-kube-api-access-zkcpj\") pod \"85f4c6ca-d293-4bfe-b736-888f35359964\" (UID: \"85f4c6ca-d293-4bfe-b736-888f35359964\") " Nov 03 00:00:19.198984 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.198862 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f4c6ca-d293-4bfe-b736-888f35359964-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f4c6ca-d293-4bfe-b736-888f35359964-maas-api-service-ca\") pod \"85f4c6ca-d293-4bfe-b736-888f35359964\" (UID: \"85f4c6ca-d293-4bfe-b736-888f35359964\") " Nov 03 00:00:19.199913 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.199556 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f4c6ca-d293-4bfe-b736-888f35359964-maas-api-service-ca" pod "85f4c6ca-d293-4bfe-b736-888f35359964" (UID: "85f4c6ca-d293-4bfe-b736-888f35359964"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:19.201697 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.201671 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f4c6ca-d293-4bfe-b736-888f35359964-kube-api-access-zkcpj" pod "85f4c6ca-d293-4bfe-b736-888f35359964" (UID: "85f4c6ca-d293-4bfe-b736-888f35359964"). InnerVolumeSpecName "kube-api-access-zkcpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:19.218229 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.218204 5672 generic.go:356] "Generic (PLEG): container finished" podID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerID="205bff68ca9a2c2a6b1639a4396a1a4c574eb1387f2e846b28beef7a98e61f54" exitCode=60 Nov 03 00:00:19.218355 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.218270 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" event={"ID":"8c2c9c15-097d-4fc2-a471-deb4c54dad7c","Type":"ContainerDied","Data":"205bff68ca9a2c2a6b1639a4396a1a4c574eb1387f2e846b28beef7a98e61f54"} Nov 03 00:00:19.218355 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.218348 5672 scope.go:122] "RemoveContainer" containerID="523dad71bb2da36635b6887399deac20ea1252064a70ecf5d6c88b79fab3bf68" Nov 03 00:00:19.218895 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.218817 5672 scope.go:122] "RemoveContainer" containerID="205bff68ca9a2c2a6b1639a4396a1a4c574eb1387f2e846b28beef7a98e61f54" Nov 03 00:00:19.219189 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:19.219122 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg_odh-ai-gateway-infra(8c2c9c15-097d-4fc2-a471-deb4c54dad7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" Nov 03 00:00:19.220053 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.220025 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" event={"ID":"c6b2d887-1c8e-4028-98be-5c90ee7bb36d","Type":"ContainerDied","Data":"74dac4a8eb1f3e89c6074af4a59ea802e570b9f2f100f4dd3a84619f756e1159"} Nov 03 00:00:19.220053 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.220040 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b" Nov 03 00:00:19.221901 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.221879 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" Nov 03 00:00:19.222864 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.222468 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4" event={"ID":"85f4c6ca-d293-4bfe-b736-888f35359964","Type":"ContainerDied","Data":"5fc68c60d86540839fe23394bd37644f267e283ac826278b53751bfc511cbf45"} Nov 03 00:00:19.224321 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.224299 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" Nov 03 00:00:19.224500 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.224476 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj" event={"ID":"5610bb86-f95e-4ee3-81db-63edc417a640","Type":"ContainerDied","Data":"4101f0daa079ae31457b19ed49dd4ba2975672b3c958f6cf1c3e61827ec327b7"} Nov 03 00:00:19.226773 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.226749 5672 generic.go:356] "Generic (PLEG): container finished" podID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerID="d8f90971659a969be22c9736a8abe1cd671dfbe6acc68dae82297016df336e03" exitCode=60 Nov 03 00:00:19.226890 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.226789 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" event={"ID":"d5dee99d-d0f4-4e91-abdf-b21758d550d4","Type":"ContainerDied","Data":"d8f90971659a969be22c9736a8abe1cd671dfbe6acc68dae82297016df336e03"} Nov 03 00:00:19.227466 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.227448 5672 scope.go:122] "RemoveContainer" containerID="d8f90971659a969be22c9736a8abe1cd671dfbe6acc68dae82297016df336e03" Nov 03 00:00:19.227736 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:19.227715 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf_odh-ai-gateway-infra(d5dee99d-d0f4-4e91-abdf-b21758d550d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" Nov 03 00:00:19.228780 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.228686 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" Nov 03 00:00:19.228780 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.228725 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b" event={"ID":"79a2752d-fd8e-4438-87b1-ade1d475dc9a","Type":"ContainerDied","Data":"1938bdcbad703b1091ad453e79783c8338b27e77ea035b15a7563c240afd2c35"} Nov 03 00:00:19.233480 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.232706 5672 generic.go:356] "Generic (PLEG): container finished" podID="295e9734-7b84-4a80-a5d7-11d68964f215" containerID="a0035c6792047c3675a0bf0b75a4aec84f0008136fddd8414454c154663d6c30" exitCode=60 Nov 03 00:00:19.233480 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.232817 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" event={"ID":"295e9734-7b84-4a80-a5d7-11d68964f215","Type":"ContainerDied","Data":"a0035c6792047c3675a0bf0b75a4aec84f0008136fddd8414454c154663d6c30"} Nov 03 00:00:19.233480 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.233142 5672 scope.go:122] "RemoveContainer" containerID="a0035c6792047c3675a0bf0b75a4aec84f0008136fddd8414454c154663d6c30" Nov 03 00:00:19.233480 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:19.233396 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm_odh-ai-gateway-infra(295e9734-7b84-4a80-a5d7-11d68964f215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" Nov 03 00:00:19.235593 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.235554 5672 generic.go:356] "Generic (PLEG): container finished" podID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerID="e3e00128dffb756756964131c094061e518852b8f1ab8dd54cdc6b2454d8f787" exitCode=60 Nov 03 00:00:19.235701 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.235642 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" event={"ID":"7131cd26-6c33-4d0f-a629-d390bc70b0c2","Type":"ContainerDied","Data":"e3e00128dffb756756964131c094061e518852b8f1ab8dd54cdc6b2454d8f787"} Nov 03 00:00:19.235958 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.235929 5672 scope.go:122] "RemoveContainer" containerID="e3e00128dffb756756964131c094061e518852b8f1ab8dd54cdc6b2454d8f787" Nov 03 00:00:19.236174 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:19.236126 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm_odh-ai-gateway-infra(7131cd26-6c33-4d0f-a629-d390bc70b0c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" Nov 03 00:00:19.239334 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.239317 5672 generic.go:356] "Generic (PLEG): container finished" podID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerID="a6145e79d1f624acb57902522c09d44fb202ed1ac856923c3103bbff7cbb3b25" exitCode=60 Nov 03 00:00:19.239417 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.239399 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" event={"ID":"2cf3fac2-f23d-49a6-b27a-b395e1722d75","Type":"ContainerDied","Data":"a6145e79d1f624acb57902522c09d44fb202ed1ac856923c3103bbff7cbb3b25"} Nov 03 00:00:19.239731 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.239712 5672 scope.go:122] "RemoveContainer" containerID="a6145e79d1f624acb57902522c09d44fb202ed1ac856923c3103bbff7cbb3b25" Nov 03 00:00:19.240112 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:19.239978 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz_odh-ai-gateway-infra(2cf3fac2-f23d-49a6-b27a-b395e1722d75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" Nov 03 00:00:19.241307 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.241285 5672 generic.go:356] "Generic (PLEG): container finished" podID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerID="24fbbfa7b03137cd323dfdf1009fbf5697102d198788ddd253857e6c020c3fba" exitCode=60 Nov 03 00:00:19.241381 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.241346 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" event={"ID":"513fd003-68d9-4e05-bf59-3c3369882a4b","Type":"ContainerDied","Data":"24fbbfa7b03137cd323dfdf1009fbf5697102d198788ddd253857e6c020c3fba"} Nov 03 00:00:19.241743 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.241725 5672 scope.go:122] "RemoveContainer" containerID="24fbbfa7b03137cd323dfdf1009fbf5697102d198788ddd253857e6c020c3fba" Nov 03 00:00:19.241953 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:19.241934 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25_odh-ai-gateway-infra(513fd003-68d9-4e05-bf59-3c3369882a4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" Nov 03 00:00:19.243062 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.242918 5672 scope.go:122] "RemoveContainer" containerID="4bad2461dc3feeead1e4559bb77895e8bd2dc22ceb3a642c345a5886ac814663" Nov 03 00:00:19.254027 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.254007 5672 scope.go:122] "RemoveContainer" containerID="b3f9510708bc87792925a95f196d794041b4bb046620b2705090f129f6871b58" Nov 03 00:00:19.265350 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.265307 5672 scope.go:122] "RemoveContainer" containerID="b22aed0460d955c8a269e118026c987a0a8014ba73ba99d3f7c2377ac80cb5be" Nov 03 00:00:19.274105 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.274088 5672 scope.go:122] "RemoveContainer" containerID="5259e449e01c3e19f8c155c27b59c4635b1ce5639938d68dc624a0d3b591b2c1" Nov 03 00:00:19.286370 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.286351 5672 scope.go:122] "RemoveContainer" containerID="1ab1bef7e781d12806cf10bc67711cc2527aa714ccaf63dc5b75cc513aacf133" Nov 03 00:00:19.294576 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.294561 5672 scope.go:122] "RemoveContainer" containerID="e6319d8a64c19cd5cf36bff81c342688e5d139379652257422aab51c0b4d7873" Nov 03 00:00:19.301645 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.301603 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkcpj\" (UniqueName: \"kubernetes.io/projected/85f4c6ca-d293-4bfe-b736-888f35359964-kube-api-access-zkcpj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.301645 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.301627 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f4c6ca-d293-4bfe-b736-888f35359964-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:19.306563 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.306545 5672 scope.go:122] "RemoveContainer" containerID="3fd4e9869e87789fb1f9df4341bab48bf84343f3aa990decf0e0f9f9c4ba5834" Nov 03 00:00:19.318520 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.318499 5672 scope.go:122] "RemoveContainer" containerID="324298c0d5fa3f300504597d436c8e0e4ea7ae0f95db2a50baf51c0b9dc0db3b" Nov 03 00:00:19.334589 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.334536 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm"] Nov 03 00:00:19.335107 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.335026 5672 scope.go:122] "RemoveContainer" containerID="c36dbd8e1398f7316c78b0710c1f7f20b12454bb9f1596a8eef3244514b2082e" Nov 03 00:00:19.422926 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.422885 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25"] Nov 03 00:00:19.529208 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.529101 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg"] Nov 03 00:00:19.542307 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.542279 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4"] Nov 03 00:00:19.545670 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.545646 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1be81-29894400-xh7g4"] Nov 03 00:00:19.861376 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.861343 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj"] Nov 03 00:00:19.864678 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:19.864642 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894400-qdwlj"] Nov 03 00:00:20.175235 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.175128 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b"] Nov 03 00:00:20.181062 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.181036 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894400-8jd5b"] Nov 03 00:00:20.282756 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.282723 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b"] Nov 03 00:00:20.294411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.290751 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894400-45d9b"] Nov 03 00:00:20.473632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.473605 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" Nov 03 00:00:20.515957 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.515926 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/513fd003-68d9-4e05-bf59-3c3369882a4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513fd003-68d9-4e05-bf59-3c3369882a4b-maas-api-service-ca\") pod \"513fd003-68d9-4e05-bf59-3c3369882a4b\" (UID: \"513fd003-68d9-4e05-bf59-3c3369882a4b\") " Nov 03 00:00:20.516114 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.516053 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513fd003-68d9-4e05-bf59-3c3369882a4b-kube-api-access-xwzzf\" (UniqueName: \"kubernetes.io/projected/513fd003-68d9-4e05-bf59-3c3369882a4b-kube-api-access-xwzzf\") pod \"513fd003-68d9-4e05-bf59-3c3369882a4b\" (UID: \"513fd003-68d9-4e05-bf59-3c3369882a4b\") " Nov 03 00:00:20.516349 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.516322 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513fd003-68d9-4e05-bf59-3c3369882a4b-maas-api-service-ca" pod "513fd003-68d9-4e05-bf59-3c3369882a4b" (UID: "513fd003-68d9-4e05-bf59-3c3369882a4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:20.516565 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.516546 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513fd003-68d9-4e05-bf59-3c3369882a4b-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.518553 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.518529 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513fd003-68d9-4e05-bf59-3c3369882a4b-kube-api-access-xwzzf" pod "513fd003-68d9-4e05-bf59-3c3369882a4b" (UID: "513fd003-68d9-4e05-bf59-3c3369882a4b"). InnerVolumeSpecName "kube-api-access-xwzzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:20.524135 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.523908 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" Nov 03 00:00:20.527637 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.527617 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" Nov 03 00:00:20.617675 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.617645 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7131cd26-6c33-4d0f-a629-d390bc70b0c2-kube-api-access-hczwr\" (UniqueName: \"kubernetes.io/projected/7131cd26-6c33-4d0f-a629-d390bc70b0c2-kube-api-access-hczwr\") pod \"7131cd26-6c33-4d0f-a629-d390bc70b0c2\" (UID: \"7131cd26-6c33-4d0f-a629-d390bc70b0c2\") " Nov 03 00:00:20.617815 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.617718 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-maas-api-service-ca\") pod \"8c2c9c15-097d-4fc2-a471-deb4c54dad7c\" (UID: \"8c2c9c15-097d-4fc2-a471-deb4c54dad7c\") " Nov 03 00:00:20.617889 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.617875 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-kube-api-access-vmltx\" (UniqueName: \"kubernetes.io/projected/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-kube-api-access-vmltx\") pod \"8c2c9c15-097d-4fc2-a471-deb4c54dad7c\" (UID: \"8c2c9c15-097d-4fc2-a471-deb4c54dad7c\") " Nov 03 00:00:20.617948 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.617897 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7131cd26-6c33-4d0f-a629-d390bc70b0c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7131cd26-6c33-4d0f-a629-d390bc70b0c2-maas-api-service-ca\") pod \"7131cd26-6c33-4d0f-a629-d390bc70b0c2\" (UID: \"7131cd26-6c33-4d0f-a629-d390bc70b0c2\") " Nov 03 00:00:20.618115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.618093 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-maas-api-service-ca" pod "8c2c9c15-097d-4fc2-a471-deb4c54dad7c" (UID: "8c2c9c15-097d-4fc2-a471-deb4c54dad7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:20.618331 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.618298 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7131cd26-6c33-4d0f-a629-d390bc70b0c2-maas-api-service-ca" pod "7131cd26-6c33-4d0f-a629-d390bc70b0c2" (UID: "7131cd26-6c33-4d0f-a629-d390bc70b0c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:20.618331 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.618319 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.618453 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.618345 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwzzf\" (UniqueName: \"kubernetes.io/projected/513fd003-68d9-4e05-bf59-3c3369882a4b-kube-api-access-xwzzf\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.619741 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.619720 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7131cd26-6c33-4d0f-a629-d390bc70b0c2-kube-api-access-hczwr" pod "7131cd26-6c33-4d0f-a629-d390bc70b0c2" (UID: "7131cd26-6c33-4d0f-a629-d390bc70b0c2"). InnerVolumeSpecName "kube-api-access-hczwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:20.620006 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.619982 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-kube-api-access-vmltx" pod "8c2c9c15-097d-4fc2-a471-deb4c54dad7c" (UID: "8c2c9c15-097d-4fc2-a471-deb4c54dad7c"). InnerVolumeSpecName "kube-api-access-vmltx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:20.659790 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.659764 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm"] Nov 03 00:00:20.716619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.715489 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" path="/var/lib/kubelet/pods/5610bb86-f95e-4ee3-81db-63edc417a640/volumes" Nov 03 00:00:20.717995 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.717573 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" path="/var/lib/kubelet/pods/79a2752d-fd8e-4438-87b1-ade1d475dc9a/volumes" Nov 03 00:00:20.718768 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.718729 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" path="/var/lib/kubelet/pods/85f4c6ca-d293-4bfe-b736-888f35359964/volumes" Nov 03 00:00:20.724879 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.721250 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" path="/var/lib/kubelet/pods/c6b2d887-1c8e-4028-98be-5c90ee7bb36d/volumes" Nov 03 00:00:20.724879 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.722598 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmltx\" (UniqueName: \"kubernetes.io/projected/8c2c9c15-097d-4fc2-a471-deb4c54dad7c-kube-api-access-vmltx\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.724879 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.722624 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7131cd26-6c33-4d0f-a629-d390bc70b0c2-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.724879 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.722642 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hczwr\" (UniqueName: \"kubernetes.io/projected/7131cd26-6c33-4d0f-a629-d390bc70b0c2-kube-api-access-hczwr\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:20.768102 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.768071 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf"] Nov 03 00:00:20.881961 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.881931 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" Nov 03 00:00:20.927933 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.927899 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295e9734-7b84-4a80-a5d7-11d68964f215-kube-api-access-hxsjx\" (UniqueName: \"kubernetes.io/projected/295e9734-7b84-4a80-a5d7-11d68964f215-kube-api-access-hxsjx\") pod \"295e9734-7b84-4a80-a5d7-11d68964f215\" (UID: \"295e9734-7b84-4a80-a5d7-11d68964f215\") " Nov 03 00:00:20.928064 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.927965 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295e9734-7b84-4a80-a5d7-11d68964f215-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295e9734-7b84-4a80-a5d7-11d68964f215-maas-api-service-ca\") pod \"295e9734-7b84-4a80-a5d7-11d68964f215\" (UID: \"295e9734-7b84-4a80-a5d7-11d68964f215\") " Nov 03 00:00:20.928643 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.928601 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295e9734-7b84-4a80-a5d7-11d68964f215-maas-api-service-ca" pod "295e9734-7b84-4a80-a5d7-11d68964f215" (UID: "295e9734-7b84-4a80-a5d7-11d68964f215"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:20.929980 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.929956 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295e9734-7b84-4a80-a5d7-11d68964f215-kube-api-access-hxsjx" pod "295e9734-7b84-4a80-a5d7-11d68964f215" (UID: "295e9734-7b84-4a80-a5d7-11d68964f215"). InnerVolumeSpecName "kube-api-access-hxsjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:20.949243 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:20.949214 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" Nov 03 00:00:21.029121 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.029082 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5dee99d-d0f4-4e91-abdf-b21758d550d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5dee99d-d0f4-4e91-abdf-b21758d550d4-maas-api-service-ca\") pod \"d5dee99d-d0f4-4e91-abdf-b21758d550d4\" (UID: \"d5dee99d-d0f4-4e91-abdf-b21758d550d4\") " Nov 03 00:00:21.029324 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.029291 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dee99d-d0f4-4e91-abdf-b21758d550d4-kube-api-access-vsqlw\" (UniqueName: \"kubernetes.io/projected/d5dee99d-d0f4-4e91-abdf-b21758d550d4-kube-api-access-vsqlw\") pod \"d5dee99d-d0f4-4e91-abdf-b21758d550d4\" (UID: \"d5dee99d-d0f4-4e91-abdf-b21758d550d4\") " Nov 03 00:00:21.029492 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.029466 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5dee99d-d0f4-4e91-abdf-b21758d550d4-maas-api-service-ca" pod "d5dee99d-d0f4-4e91-abdf-b21758d550d4" (UID: "d5dee99d-d0f4-4e91-abdf-b21758d550d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:21.029720 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.029699 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5dee99d-d0f4-4e91-abdf-b21758d550d4-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:21.029786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.029731 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxsjx\" (UniqueName: \"kubernetes.io/projected/295e9734-7b84-4a80-a5d7-11d68964f215-kube-api-access-hxsjx\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:21.029786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.029746 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295e9734-7b84-4a80-a5d7-11d68964f215-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:21.031628 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.031604 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dee99d-d0f4-4e91-abdf-b21758d550d4-kube-api-access-vsqlw" pod "d5dee99d-d0f4-4e91-abdf-b21758d550d4" (UID: "d5dee99d-d0f4-4e91-abdf-b21758d550d4"). InnerVolumeSpecName "kube-api-access-vsqlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:21.082678 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.082649 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz"] Nov 03 00:00:21.131434 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.131397 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsqlw\" (UniqueName: \"kubernetes.io/projected/d5dee99d-d0f4-4e91-abdf-b21758d550d4-kube-api-access-vsqlw\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:21.211947 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.211922 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" Nov 03 00:00:21.285118 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.285037 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" event={"ID":"8c2c9c15-097d-4fc2-a471-deb4c54dad7c","Type":"ContainerDied","Data":"0ae587d8e97a8ba399323265704a1fb8d62b1f1373e5635aca63037ab56c90c3"} Nov 03 00:00:21.285679 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.285642 5672 scope.go:122] "RemoveContainer" containerID="205bff68ca9a2c2a6b1639a4396a1a4c574eb1387f2e846b28beef7a98e61f54" Nov 03 00:00:21.285857 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.285075 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg" Nov 03 00:00:21.287478 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.287456 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" Nov 03 00:00:21.287613 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.287480 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf" event={"ID":"d5dee99d-d0f4-4e91-abdf-b21758d550d4","Type":"ContainerDied","Data":"3ec726ac20212c23bf312ec4ed12d637898666e37cddfff67bfc70194447b9fa"} Nov 03 00:00:21.297189 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.297170 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" Nov 03 00:00:21.297335 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.297308 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm" event={"ID":"295e9734-7b84-4a80-a5d7-11d68964f215","Type":"ContainerDied","Data":"5b55bac55c5314c13f785a7c7dce4ac93f7f22a4885011278d659b9e1cfd1d28"} Nov 03 00:00:21.299384 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.299369 5672 scope.go:122] "RemoveContainer" containerID="d8f90971659a969be22c9736a8abe1cd671dfbe6acc68dae82297016df336e03" Nov 03 00:00:21.299454 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.299394 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" event={"ID":"7131cd26-6c33-4d0f-a629-d390bc70b0c2","Type":"ContainerDied","Data":"0244df101ab071b0df5edd1cf5d7e4f384a5a3dd5c80e2c582412200f8d6fdc8"} Nov 03 00:00:21.299513 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.299480 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm" Nov 03 00:00:21.301434 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.301390 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" event={"ID":"2cf3fac2-f23d-49a6-b27a-b395e1722d75","Type":"ContainerDied","Data":"d29b74450a8f6d562c0df11aab61d31bb23e73305630a7f5e6a688779e9c03aa"} Nov 03 00:00:21.301537 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.301482 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz" Nov 03 00:00:21.303197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.303136 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" event={"ID":"513fd003-68d9-4e05-bf59-3c3369882a4b","Type":"ContainerDied","Data":"d22cdfff513030afc6441ef9f4553b29af7481b19446ef3bbd1a9859396d1c74"} Nov 03 00:00:21.303291 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.303280 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25" Nov 03 00:00:21.311716 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.311693 5672 scope.go:122] "RemoveContainer" containerID="a0035c6792047c3675a0bf0b75a4aec84f0008136fddd8414454c154663d6c30" Nov 03 00:00:21.322175 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.322137 5672 scope.go:122] "RemoveContainer" containerID="e3e00128dffb756756964131c094061e518852b8f1ab8dd54cdc6b2454d8f787" Nov 03 00:00:21.330326 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.330310 5672 scope.go:122] "RemoveContainer" containerID="a6145e79d1f624acb57902522c09d44fb202ed1ac856923c3103bbff7cbb3b25" Nov 03 00:00:21.332516 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.332498 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf3fac2-f23d-49a6-b27a-b395e1722d75-kube-api-access-7qtjp\" (UniqueName: \"kubernetes.io/projected/2cf3fac2-f23d-49a6-b27a-b395e1722d75-kube-api-access-7qtjp\") pod \"2cf3fac2-f23d-49a6-b27a-b395e1722d75\" (UID: \"2cf3fac2-f23d-49a6-b27a-b395e1722d75\") " Nov 03 00:00:21.332673 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.332662 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf3fac2-f23d-49a6-b27a-b395e1722d75-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3fac2-f23d-49a6-b27a-b395e1722d75-maas-api-service-ca\") pod \"2cf3fac2-f23d-49a6-b27a-b395e1722d75\" (UID: \"2cf3fac2-f23d-49a6-b27a-b395e1722d75\") " Nov 03 00:00:21.333077 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.333050 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf3fac2-f23d-49a6-b27a-b395e1722d75-maas-api-service-ca" pod "2cf3fac2-f23d-49a6-b27a-b395e1722d75" (UID: "2cf3fac2-f23d-49a6-b27a-b395e1722d75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:00:21.334472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.334440 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf3fac2-f23d-49a6-b27a-b395e1722d75-kube-api-access-7qtjp" pod "2cf3fac2-f23d-49a6-b27a-b395e1722d75" (UID: "2cf3fac2-f23d-49a6-b27a-b395e1722d75"). InnerVolumeSpecName "kube-api-access-7qtjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:00:21.339102 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.339079 5672 scope.go:122] "RemoveContainer" containerID="24fbbfa7b03137cd323dfdf1009fbf5697102d198788ddd253857e6c020c3fba" Nov 03 00:00:21.433986 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.433959 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qtjp\" (UniqueName: \"kubernetes.io/projected/2cf3fac2-f23d-49a6-b27a-b395e1722d75-kube-api-access-7qtjp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:21.433986 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.433984 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3fac2-f23d-49a6-b27a-b395e1722d75-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:00:21.677335 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.677303 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm"] Nov 03 00:00:21.683026 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.682996 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894400-6rnxm"] Nov 03 00:00:21.784251 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.784220 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf"] Nov 03 00:00:21.791511 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.791485 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894400-b8ckf"] Nov 03 00:00:21.892228 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.892161 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm"] Nov 03 00:00:21.895917 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:21.895891 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894400-qlwrm"] Nov 03 00:00:22.045120 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.045087 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25"] Nov 03 00:00:22.050709 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.050682 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b4d53e-29894400-4wt25"] Nov 03 00:00:22.152947 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.152870 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg"] Nov 03 00:00:22.157979 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.157957 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894400-knttg"] Nov 03 00:00:22.257503 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.257469 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz"] Nov 03 00:00:22.261345 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.261319 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894400-9bzpz"] Nov 03 00:00:22.713008 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.712966 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" path="/var/lib/kubelet/pods/295e9734-7b84-4a80-a5d7-11d68964f215/volumes" Nov 03 00:00:22.713597 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.713576 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" path="/var/lib/kubelet/pods/2cf3fac2-f23d-49a6-b27a-b395e1722d75/volumes" Nov 03 00:00:22.714022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.714000 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" path="/var/lib/kubelet/pods/513fd003-68d9-4e05-bf59-3c3369882a4b/volumes" Nov 03 00:00:22.714759 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.714737 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" path="/var/lib/kubelet/pods/7131cd26-6c33-4d0f-a629-d390bc70b0c2/volumes" Nov 03 00:00:22.715181 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.715163 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" path="/var/lib/kubelet/pods/8c2c9c15-097d-4fc2-a471-deb4c54dad7c/volumes" Nov 03 00:00:22.715579 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:22.715561 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" path="/var/lib/kubelet/pods/d5dee99d-d0f4-4e91-abdf-b21758d550d4/volumes" Nov 03 00:00:26.052341 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:26.052297 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:00:33.634213 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634170 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl"] Nov 03 00:00:33.634803 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634749 5672 cpu_manager.go:414] "containerMap: removing container" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" containerName="cleanup" Nov 03 00:00:33.634803 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634767 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" containerName="cleanup" Nov 03 00:00:33.634803 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634780 5672 cpu_manager.go:414] "containerMap: removing container" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" containerName="cleanup" Nov 03 00:00:33.634803 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634785 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" containerName="cleanup" Nov 03 00:00:33.634803 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634793 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerName="cleanup" Nov 03 00:00:33.634803 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634800 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634811 5672 cpu_manager.go:414] "containerMap: removing container" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634815 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634822 5672 cpu_manager.go:414] "containerMap: removing container" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634827 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634836 5672 cpu_manager.go:414] "containerMap: removing container" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634841 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634846 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634851 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634864 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634870 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634879 5672 cpu_manager.go:414] "containerMap: removing container" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634883 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634888 5672 cpu_manager.go:414] "containerMap: removing container" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634893 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634902 5672 cpu_manager.go:414] "containerMap: removing container" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634907 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634915 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634919 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634925 5672 cpu_manager.go:414] "containerMap: removing container" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634930 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634937 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634942 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634947 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634952 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634962 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634967 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634972 5672 cpu_manager.go:414] "containerMap: removing container" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634976 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634983 5672 cpu_manager.go:414] "containerMap: removing container" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.634987 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635004 5672 cpu_manager.go:414] "containerMap: removing container" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635009 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635018 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635022 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635027 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635032 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635038 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635043 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635054 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635058 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635064 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635068 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635075 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635079 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635169 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635177 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" containerName="cleanup" Nov 03 00:00:33.635140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635184 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635191 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635196 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635202 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635208 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635216 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635225 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635230 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635237 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635243 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635248 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635253 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635258 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635264 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635269 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635276 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635281 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635285 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635289 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635295 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635300 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635305 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.635310 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2c9c15-097d-4fc2-a471-deb4c54dad7c" containerName="cleanup" Nov 03 00:00:33.638002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.638000 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:00:33.691239 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.691208 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl"] Nov 03 00:00:33.706848 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.706817 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl\" (UID: \"6c95b71b-5c1a-43ec-8314-8c868fdf4a64\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:00:33.706999 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.706953 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszsj\" (UniqueName: \"kubernetes.io/projected/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-kube-api-access-zszsj\") pod \"payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl\" (UID: \"6c95b71b-5c1a-43ec-8314-8c868fdf4a64\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:00:33.734106 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.734077 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-666a38\"" Nov 03 00:00:33.734106 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.734092 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-666a38-dockercfg-62m5c\"" Nov 03 00:00:33.807849 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.807813 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl\" (UID: \"6c95b71b-5c1a-43ec-8314-8c868fdf4a64\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:00:33.808060 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.807890 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszsj\" (UniqueName: \"kubernetes.io/projected/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-kube-api-access-zszsj\") pod \"payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl\" (UID: \"6c95b71b-5c1a-43ec-8314-8c868fdf4a64\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:00:33.808619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.808597 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl\" (UID: \"6c95b71b-5c1a-43ec-8314-8c868fdf4a64\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:00:33.861794 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.861766 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszsj\" (UniqueName: \"kubernetes.io/projected/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-kube-api-access-zszsj\") pod \"payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl\" (UID: \"6c95b71b-5c1a-43ec-8314-8c868fdf4a64\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:00:33.951258 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:33.951165 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:00:34.318701 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:34.318675 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl"] Nov 03 00:00:34.320098 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:00:34.320070 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c95b71b_5c1a_43ec_8314_8c868fdf4a64.slice/crio-60b84c654787e950cc2511c0ae30dfb9190b9e5564c95c22743f3f9e2a58b931 WatchSource:0}: Error finding container 60b84c654787e950cc2511c0ae30dfb9190b9e5564c95c22743f3f9e2a58b931: Status 404 returned error can't find the container with id 60b84c654787e950cc2511c0ae30dfb9190b9e5564c95c22743f3f9e2a58b931 Nov 03 00:00:34.401101 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:34.401071 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" event={"ID":"6c95b71b-5c1a-43ec-8314-8c868fdf4a64","Type":"ContainerStarted","Data":"3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe"} Nov 03 00:00:34.401234 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:34.401131 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" event={"ID":"6c95b71b-5c1a-43ec-8314-8c868fdf4a64","Type":"ContainerStarted","Data":"60b84c654787e950cc2511c0ae30dfb9190b9e5564c95c22743f3f9e2a58b931"} Nov 03 00:00:34.401234 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:34.401192 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:00:34.507009 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:34.506959 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" podStartSLOduration=1.5069458770000002 podStartE2EDuration="1.506945877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:34.505645881 +0000 UTC m=+4027.912957692" watchObservedRunningTime="2026-11-03 00:00:34.506945877 +0000 UTC m=+4027.914257706" Nov 03 00:00:38.894434 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.894398 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6"] Nov 03 00:00:38.894967 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.894951 5672 cpu_manager.go:414] "containerMap: removing container" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" containerName="cleanup" Nov 03 00:00:38.895013 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.894968 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" containerName="cleanup" Nov 03 00:00:38.895013 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.894997 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerName="cleanup" Nov 03 00:00:38.895013 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895003 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5dee99d-d0f4-4e91-abdf-b21758d550d4" containerName="cleanup" Nov 03 00:00:38.895115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895017 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerName="cleanup" Nov 03 00:00:38.895115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895022 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7131cd26-6c33-4d0f-a629-d390bc70b0c2" containerName="cleanup" Nov 03 00:00:38.895115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895034 5672 cpu_manager.go:414] "containerMap: removing container" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" containerName="cleanup" Nov 03 00:00:38.895115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895039 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5610bb86-f95e-4ee3-81db-63edc417a640" containerName="cleanup" Nov 03 00:00:38.895115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895047 5672 cpu_manager.go:414] "containerMap: removing container" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerName="cleanup" Nov 03 00:00:38.895115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895052 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a2752d-fd8e-4438-87b1-ade1d475dc9a" containerName="cleanup" Nov 03 00:00:38.895347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895134 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="513fd003-68d9-4e05-bf59-3c3369882a4b" containerName="cleanup" Nov 03 00:00:38.895347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895160 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b2d887-1c8e-4028-98be-5c90ee7bb36d" containerName="cleanup" Nov 03 00:00:38.895347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895171 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="295e9734-7b84-4a80-a5d7-11d68964f215" containerName="cleanup" Nov 03 00:00:38.895347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895180 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f4c6ca-d293-4bfe-b736-888f35359964" containerName="cleanup" Nov 03 00:00:38.895347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.895187 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf3fac2-f23d-49a6-b27a-b395e1722d75" containerName="cleanup" Nov 03 00:00:38.897990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.897968 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:38.950173 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.950129 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c9d4fc-openshift-default-dockercfg-drxl2\"" Nov 03 00:00:38.999178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.962317 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istiod-ca-cert\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:38.999178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.983720 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-credential-socket\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:38.999178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.983785 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-podinfo\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:38.999178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.983822 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-certs\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:38.999178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.983851 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-socket\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:38.999178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.983930 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqbzx\" (UniqueName: \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-kube-api-access-gqbzx\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:38.999178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.983973 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-data\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:38.999178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.984095 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-envoy\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:38.999178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:38.984122 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-token\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.018668 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.017207 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6"] Nov 03 00:00:39.090671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.084948 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-envoy\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.090671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.084995 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-token\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.090671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.085117 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istiod-ca-cert\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.090671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.085169 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-credential-socket\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.090671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.085208 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-podinfo\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.090671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.085239 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-certs\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.090671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.085268 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-socket\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.090671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.085333 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqbzx\" (UniqueName: \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-kube-api-access-gqbzx\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.090671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.085367 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-data\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.090671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.085898 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-data\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.090671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.086122 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-credential-socket\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.091475 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.091111 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-certs\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.091843 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.091820 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-socket\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.092945 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.092921 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istiod-ca-cert\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.093929 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.093903 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-podinfo\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.096373 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.096332 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-envoy\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.145030 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.144957 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-token\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.145334 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.145308 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqbzx\" (UniqueName: \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-kube-api-access-gqbzx\") pod \"e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.211713 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.211679 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:39.644314 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.644284 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6"] Nov 03 00:00:39.646565 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:00:39.646535 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73f440b_eeec_4ce7_861c_3171d6f6d70b.slice/crio-fd2ca2ffd3b5a0c60f40126e45a93fb91729dfcd81b75a76b6f1731c037fa1c4 WatchSource:0}: Error finding container fd2ca2ffd3b5a0c60f40126e45a93fb91729dfcd81b75a76b6f1731c037fa1c4: Status 404 returned error can't find the container with id fd2ca2ffd3b5a0c60f40126e45a93fb91729dfcd81b75a76b6f1731c037fa1c4 Nov 03 00:00:39.649671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.649635 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:39.649756 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.649721 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:39.649804 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:39.649763 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:00:40.444498 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:40.444464 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" event={"ID":"c73f440b-eeec-4ce7-861c-3171d6f6d70b","Type":"ContainerStarted","Data":"b3fc4e80973e01db9e5758cb45500adc62bcd36a55b31fd982286b8067ca6187"} Nov 03 00:00:40.445010 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:40.444546 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" event={"ID":"c73f440b-eeec-4ce7-861c-3171d6f6d70b","Type":"ContainerStarted","Data":"fd2ca2ffd3b5a0c60f40126e45a93fb91729dfcd81b75a76b6f1731c037fa1c4"} Nov 03 00:00:40.603025 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:40.602974 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" podStartSLOduration=2.602959511 podStartE2EDuration="2.602959511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:40.600814838 +0000 UTC m=+4034.008126670" watchObservedRunningTime="2026-11-03 00:00:40.602959511 +0000 UTC m=+4034.010271340" Nov 03 00:00:41.211887 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:41.211852 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:41.217540 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:41.217512 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:41.451523 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:41.451489 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:41.452914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:41.452887 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:00:45.410037 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:45.410005 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:00:49.453976 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.453943 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq"] Nov 03 00:00:49.458591 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.458568 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:00:49.510246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.510214 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq"] Nov 03 00:00:49.548141 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.548107 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c9d4fc\"" Nov 03 00:00:49.551802 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.551773 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c9d4fc-dockercfg-ncqdc\"" Nov 03 00:00:49.616350 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.616323 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gq4z\" (UniqueName: \"kubernetes.io/projected/17169a76-b5d3-4bde-b337-64ac068c8798-kube-api-access-8gq4z\") pod \"payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq\" (UID: \"17169a76-b5d3-4bde-b337-64ac068c8798\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:00:49.616493 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.616355 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17169a76-b5d3-4bde-b337-64ac068c8798-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq\" (UID: \"17169a76-b5d3-4bde-b337-64ac068c8798\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:00:49.717442 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.717370 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gq4z\" (UniqueName: \"kubernetes.io/projected/17169a76-b5d3-4bde-b337-64ac068c8798-kube-api-access-8gq4z\") pod \"payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq\" (UID: \"17169a76-b5d3-4bde-b337-64ac068c8798\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:00:49.717442 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.717405 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17169a76-b5d3-4bde-b337-64ac068c8798-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq\" (UID: \"17169a76-b5d3-4bde-b337-64ac068c8798\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:00:49.718116 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.718098 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17169a76-b5d3-4bde-b337-64ac068c8798-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq\" (UID: \"17169a76-b5d3-4bde-b337-64ac068c8798\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:00:49.771061 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.771034 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gq4z\" (UniqueName: \"kubernetes.io/projected/17169a76-b5d3-4bde-b337-64ac068c8798-kube-api-access-8gq4z\") pod \"payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq\" (UID: \"17169a76-b5d3-4bde-b337-64ac068c8798\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:00:49.773906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:49.773885 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:00:50.149429 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:50.149396 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq"] Nov 03 00:00:50.149584 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:00:50.149514 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17169a76_b5d3_4bde_b337_64ac068c8798.slice/crio-459bc41e8c4c45b94c095c5de46c074841941b92b23545b01937f9039edb59f6 WatchSource:0}: Error finding container 459bc41e8c4c45b94c095c5de46c074841941b92b23545b01937f9039edb59f6: Status 404 returned error can't find the container with id 459bc41e8c4c45b94c095c5de46c074841941b92b23545b01937f9039edb59f6 Nov 03 00:00:50.512308 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:50.512278 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" event={"ID":"17169a76-b5d3-4bde-b337-64ac068c8798","Type":"ContainerStarted","Data":"4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f"} Nov 03 00:00:50.512733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:50.512349 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" event={"ID":"17169a76-b5d3-4bde-b337-64ac068c8798","Type":"ContainerStarted","Data":"459bc41e8c4c45b94c095c5de46c074841941b92b23545b01937f9039edb59f6"} Nov 03 00:00:50.512733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:50.512491 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:00:50.621171 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:00:50.621089 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" podStartSLOduration=1.621075774 podStartE2EDuration="1.621075774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:50.617938847 +0000 UTC m=+4044.025250675" watchObservedRunningTime="2026-11-03 00:00:50.621075774 +0000 UTC m=+4044.028387602" Nov 03 00:00:56.021083 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:00:56.021047 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:01:01.523435 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:01.523402 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:01:05.635030 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:05.634956 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:01:16.739480 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:16.739439 5672 generic.go:356] "Generic (PLEG): container finished" podID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerID="5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe" exitCode=0 Nov 03 00:01:16.740031 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:16.739511 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" event={"ID":"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0","Type":"ContainerDied","Data":"5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe"} Nov 03 00:01:16.740031 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:16.740022 5672 scope.go:122] "RemoveContainer" containerID="5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe" Nov 03 00:01:16.741693 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:16.741664 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:01:16.741800 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:16.741730 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:01:16.741800 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:16.741762 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:01:17.759850 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:17.759810 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" event={"ID":"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0","Type":"ContainerStarted","Data":"55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856"} Nov 03 00:01:18.360862 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:18.360821 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 03 00:01:18.361982 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:18.361956 5672 patch_prober.go:29] interesting pod/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 00:01:18.362078 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:18.362032 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 00:01:19.361663 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:19.361627 5672 patch_prober.go:29] interesting pod/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 00:01:19.362053 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:19.361693 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 00:01:20.360831 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:20.360788 5672 patch_prober.go:29] interesting pod/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 00:01:20.361064 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:20.360872 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 00:01:20.556174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:20.555307 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 03 00:01:21.360731 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:21.360690 5672 patch_prober.go:29] interesting pod/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 00:01:21.360965 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:21.360769 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 00:01:22.360712 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:22.360671 5672 patch_prober.go:29] interesting pod/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 03 00:01:22.361163 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:22.360737 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 03 00:01:23.365767 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:23.365731 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 03 00:01:23.367083 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:23.367057 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 03 00:01:26.021032 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:01:26.020989 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:01:56.021362 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:01:56.021323 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:01:57.780578 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:01:57.780541 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:02:17.236176 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.235841 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd"] Nov 03 00:02:17.255277 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.255246 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.347120 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.347079 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd"] Nov 03 00:02:17.361904 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.361873 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6d61f9-openshift-default-dockercfg-h5m4v\"" Nov 03 00:02:17.364860 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.364829 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h55ps\" (UniqueName: \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-kube-api-access-h55ps\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.365029 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.364894 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-envoy\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.365029 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.364954 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istiod-ca-cert\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.365029 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.364980 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-data\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.365029 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.365005 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-token\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.365323 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.365078 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-podinfo\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.365323 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.365109 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-credential-socket\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.365323 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.365188 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-socket\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.365323 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.365239 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-certs\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.466273 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.466238 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h55ps\" (UniqueName: \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-kube-api-access-h55ps\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.466478 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.466304 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-envoy\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.466478 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.466376 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istiod-ca-cert\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.466478 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.466403 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-data\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.466783 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.466432 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-token\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.467044 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.466795 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-data\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.467200 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.467179 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-podinfo\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.467784 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.467758 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-credential-socket\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.468139 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.468081 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-credential-socket\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.468584 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.468543 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-socket\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.468691 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.467617 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istiod-ca-cert\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.468801 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.468785 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-socket\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.468915 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.468890 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-envoy\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.469016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.469002 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-certs\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.469309 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.469285 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-certs\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.470453 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.470431 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-podinfo\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.520712 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.520641 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-token\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.522881 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.522852 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h55ps\" (UniqueName: \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-kube-api-access-h55ps\") pod \"e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:17.607515 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:17.607475 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:18.098535 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:18.098497 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd"] Nov 03 00:02:18.098705 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:02:18.098584 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29c3752_1e71_40ba_a9b2_bfeb439fef45.slice/crio-de4b8be966b05a85ec9fd0881a84022326c567e45a1e22c435d44f9166b6efab WatchSource:0}: Error finding container de4b8be966b05a85ec9fd0881a84022326c567e45a1e22c435d44f9166b6efab: Status 404 returned error can't find the container with id de4b8be966b05a85ec9fd0881a84022326c567e45a1e22c435d44f9166b6efab Nov 03 00:02:18.101610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:18.101576 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:18.101703 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:18.101645 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:18.101703 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:18.101676 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:18.186089 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:18.186053 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" event={"ID":"d29c3752-1e71-40ba-a9b2-bfeb439fef45","Type":"ContainerStarted","Data":"de4b8be966b05a85ec9fd0881a84022326c567e45a1e22c435d44f9166b6efab"} Nov 03 00:02:19.194571 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:19.194533 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" event={"ID":"d29c3752-1e71-40ba-a9b2-bfeb439fef45","Type":"ContainerStarted","Data":"fe62798bbd139345218ebad3f647a9afe00170ba5025926d5a6ad26ca2904e17"} Nov 03 00:02:19.608567 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:19.608533 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:19.615092 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:19.615061 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:19.730961 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:19.730912 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" podStartSLOduration=2.730897675 podStartE2EDuration="2.730897675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:02:19.388909484 +0000 UTC m=+4132.796221314" watchObservedRunningTime="2026-11-03 00:02:19.730897675 +0000 UTC m=+4133.138209572" Nov 03 00:02:20.201525 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:20.201488 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:20.202536 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:20.202513 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:02:25.845918 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:25.845881 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq"] Nov 03 00:02:25.849399 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:25.849379 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:02:25.951277 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:25.951246 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6d61f9-dockercfg-vgt7z\"" Nov 03 00:02:25.951277 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:25.951247 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6d61f9\"" Nov 03 00:02:25.954721 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:25.954692 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq"] Nov 03 00:02:25.988500 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:25.988460 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq\" (UID: \"db20f18a-6f58-4b0a-ac46-1c3a22b093dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:02:25.988712 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:25.988614 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8z42\" (UniqueName: \"kubernetes.io/projected/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-kube-api-access-l8z42\") pod \"payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq\" (UID: \"db20f18a-6f58-4b0a-ac46-1c3a22b093dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:02:26.039216 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.035218 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw"] Nov 03 00:02:26.054213 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.040980 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:02:26.074282 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:02:26.074241 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:02:26.079452 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.079419 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw"] Nov 03 00:02:26.090297 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.090263 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq\" (UID: \"db20f18a-6f58-4b0a-ac46-1c3a22b093dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:02:26.090496 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.090378 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8z42\" (UniqueName: \"kubernetes.io/projected/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-kube-api-access-l8z42\") pod \"payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq\" (UID: \"db20f18a-6f58-4b0a-ac46-1c3a22b093dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:02:26.091342 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.091310 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq\" (UID: \"db20f18a-6f58-4b0a-ac46-1c3a22b093dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:02:26.144485 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.144405 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8z42\" (UniqueName: \"kubernetes.io/projected/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-kube-api-access-l8z42\") pod \"payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq\" (UID: \"db20f18a-6f58-4b0a-ac46-1c3a22b093dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:02:26.163224 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.163197 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:02:26.192098 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.192065 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td254\" (UniqueName: \"kubernetes.io/projected/1c972e13-18da-4503-ad48-e54e2a31e5c9-kube-api-access-td254\") pod \"payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw\" (UID: \"1c972e13-18da-4503-ad48-e54e2a31e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:02:26.192308 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.192283 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c972e13-18da-4503-ad48-e54e2a31e5c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw\" (UID: \"1c972e13-18da-4503-ad48-e54e2a31e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:02:26.294174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.293752 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td254\" (UniqueName: \"kubernetes.io/projected/1c972e13-18da-4503-ad48-e54e2a31e5c9-kube-api-access-td254\") pod \"payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw\" (UID: \"1c972e13-18da-4503-ad48-e54e2a31e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:02:26.294174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.293869 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c972e13-18da-4503-ad48-e54e2a31e5c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw\" (UID: \"1c972e13-18da-4503-ad48-e54e2a31e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:02:26.294836 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.294806 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c972e13-18da-4503-ad48-e54e2a31e5c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw\" (UID: \"1c972e13-18da-4503-ad48-e54e2a31e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:02:26.358214 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.358177 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td254\" (UniqueName: \"kubernetes.io/projected/1c972e13-18da-4503-ad48-e54e2a31e5c9-kube-api-access-td254\") pod \"payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw\" (UID: \"1c972e13-18da-4503-ad48-e54e2a31e5c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:02:26.366626 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.366592 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:02:26.555477 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.555449 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq"] Nov 03 00:02:26.558222 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:02:26.558193 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb20f18a_6f58_4b0a_ac46_1c3a22b093dd.slice/crio-d16f7825a80aaada260fb3609c71f30b3450f5bfa96cb5d7476cde80f4a00adc WatchSource:0}: Error finding container d16f7825a80aaada260fb3609c71f30b3450f5bfa96cb5d7476cde80f4a00adc: Status 404 returned error can't find the container with id d16f7825a80aaada260fb3609c71f30b3450f5bfa96cb5d7476cde80f4a00adc Nov 03 00:02:26.790029 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.790001 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw"] Nov 03 00:02:26.791351 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:02:26.791314 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c972e13_18da_4503_ad48_e54e2a31e5c9.slice/crio-1be9afd818ea076ae700cf6bcb21b8a3af267fdf58dde2d00a60d5ed7c0e2d46 WatchSource:0}: Error finding container 1be9afd818ea076ae700cf6bcb21b8a3af267fdf58dde2d00a60d5ed7c0e2d46: Status 404 returned error can't find the container with id 1be9afd818ea076ae700cf6bcb21b8a3af267fdf58dde2d00a60d5ed7c0e2d46 Nov 03 00:02:26.837482 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.837445 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x"] Nov 03 00:02:26.843619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.843588 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:26.900087 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.900053 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-666a38-openshift-default-dockercfg-xbjck\"" Nov 03 00:02:26.901311 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:26.901275 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x"] Nov 03 00:02:27.003223 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.003182 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-token\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.003479 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.003235 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-envoy\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.003479 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.003308 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-certs\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.003479 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.003347 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671563f9-fd52-4376-9496-1bb3aa4a2254-istiod-ca-cert\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.003479 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.003365 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-data\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.003479 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.003451 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79d4\" (UniqueName: \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-kube-api-access-c79d4\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.003668 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.003503 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.003668 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.003604 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-podinfo\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.003668 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.003633 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-credential-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.104559 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.104468 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-podinfo\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.104559 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.104510 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-credential-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.104795 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.104583 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-token\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.104795 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.104611 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-envoy\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.104795 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.104643 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-certs\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.104795 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.104675 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671563f9-fd52-4376-9496-1bb3aa4a2254-istiod-ca-cert\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.104795 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.104697 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-data\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.104795 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.104753 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79d4\" (UniqueName: \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-kube-api-access-c79d4\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.105188 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.104802 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.105188 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.105060 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-credential-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.105188 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.105085 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-certs\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.105188 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.105169 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-data\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.105445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.105295 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.105761 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.105734 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671563f9-fd52-4376-9496-1bb3aa4a2254-istiod-ca-cert\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.107002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.106980 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-envoy\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.107207 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.107189 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-podinfo\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.160700 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.160633 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-token\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.160896 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.160746 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79d4\" (UniqueName: \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-kube-api-access-c79d4\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.169190 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.169128 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:27.261522 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.261488 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" event={"ID":"1c972e13-18da-4503-ad48-e54e2a31e5c9","Type":"ContainerStarted","Data":"114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190"} Nov 03 00:02:27.261710 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.261548 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" event={"ID":"1c972e13-18da-4503-ad48-e54e2a31e5c9","Type":"ContainerStarted","Data":"1be9afd818ea076ae700cf6bcb21b8a3af267fdf58dde2d00a60d5ed7c0e2d46"} Nov 03 00:02:27.262441 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.262421 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:02:27.263995 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.263969 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" event={"ID":"db20f18a-6f58-4b0a-ac46-1c3a22b093dd","Type":"ContainerStarted","Data":"78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4"} Nov 03 00:02:27.264095 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.264017 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" event={"ID":"db20f18a-6f58-4b0a-ac46-1c3a22b093dd","Type":"ContainerStarted","Data":"d16f7825a80aaada260fb3609c71f30b3450f5bfa96cb5d7476cde80f4a00adc"} Nov 03 00:02:27.264685 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.264665 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:02:27.376654 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.376532 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" podStartSLOduration=2.3765168389999998 podStartE2EDuration="2.376516839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:02:27.375483105 +0000 UTC m=+4140.782794948" watchObservedRunningTime="2026-11-03 00:02:27.376516839 +0000 UTC m=+4140.783828668" Nov 03 00:02:27.502110 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.502042 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" podStartSLOduration=2.502022964 podStartE2EDuration="2.502022964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:02:27.499080129 +0000 UTC m=+4140.906391941" watchObservedRunningTime="2026-11-03 00:02:27.502022964 +0000 UTC m=+4140.909334796" Nov 03 00:02:27.610851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.610819 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x"] Nov 03 00:02:27.615747 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:02:27.615702 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671563f9_fd52_4376_9496_1bb3aa4a2254.slice/crio-9264b0fe5032ef12f6b6c7f0c01a959c42624719bab0ca23aae5bb4acfa99e8d WatchSource:0}: Error finding container 9264b0fe5032ef12f6b6c7f0c01a959c42624719bab0ca23aae5bb4acfa99e8d: Status 404 returned error can't find the container with id 9264b0fe5032ef12f6b6c7f0c01a959c42624719bab0ca23aae5bb4acfa99e8d Nov 03 00:02:27.619493 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.619455 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:27.619643 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.619535 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:27.619643 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:27.619567 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:28.272908 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:28.272874 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" event={"ID":"671563f9-fd52-4376-9496-1bb3aa4a2254","Type":"ContainerStarted","Data":"d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d"} Nov 03 00:02:28.273441 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:28.272929 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" event={"ID":"671563f9-fd52-4376-9496-1bb3aa4a2254","Type":"ContainerStarted","Data":"9264b0fe5032ef12f6b6c7f0c01a959c42624719bab0ca23aae5bb4acfa99e8d"} Nov 03 00:02:28.387952 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:28.387885 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" podStartSLOduration=2.387863884 podStartE2EDuration="2.387863884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:02:28.384718101 +0000 UTC m=+4141.792029966" watchObservedRunningTime="2026-11-03 00:02:28.387863884 +0000 UTC m=+4141.795175715" Nov 03 00:02:29.170217 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:29.170186 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:29.171889 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:29.171862 5672 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 03 00:02:29.172018 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:29.171910 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" podUID="671563f9-fd52-4376-9496-1bb3aa4a2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 03 00:02:29.422828 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:29.422749 5672 scope.go:122] "RemoveContainer" containerID="68fa82454ba9165552a652912598512647ebc4c4794541312f5d6ba88ba5856c" Nov 03 00:02:29.457562 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:29.457535 5672 scope.go:122] "RemoveContainer" containerID="3c7dd442b20338a6c3eb48b77aa1de66788f51e1bd4f89f957a5fc1b4faf62b3" Nov 03 00:02:30.170429 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:30.170393 5672 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 03 00:02:30.170671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:30.170451 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" podUID="671563f9-fd52-4376-9496-1bb3aa4a2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 03 00:02:31.170113 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:31.170080 5672 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 03 00:02:31.170564 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:31.170172 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" podUID="671563f9-fd52-4376-9496-1bb3aa4a2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 03 00:02:32.170510 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:32.170472 5672 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 03 00:02:32.170925 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:32.170540 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" podUID="671563f9-fd52-4376-9496-1bb3aa4a2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 03 00:02:33.170656 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:33.170617 5672 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 03 00:02:33.171056 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:33.170695 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" podUID="671563f9-fd52-4376-9496-1bb3aa4a2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 03 00:02:34.174629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:34.174596 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:34.175102 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:34.174893 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:34.175638 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:34.175621 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:02:39.282568 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:39.282485 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:02:39.282568 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:39.282550 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:02:39.906321 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:39.906276 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:02:48.057752 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:48.055751 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw"] Nov 03 00:02:48.100109 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:48.100074 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e01a905-sc5pw"] Nov 03 00:02:48.715681 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:48.715642 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1663304d-daba-420d-8c7c-bd104b1ee141" path="/var/lib/kubelet/pods/1663304d-daba-420d-8c7c-bd104b1ee141/volumes" Nov 03 00:02:50.432046 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:50.432009 5672 generic.go:356] "Generic (PLEG): container finished" podID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerID="b3fc4e80973e01db9e5758cb45500adc62bcd36a55b31fd982286b8067ca6187" exitCode=0 Nov 03 00:02:50.432505 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:50.432076 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" event={"ID":"c73f440b-eeec-4ce7-861c-3171d6f6d70b","Type":"ContainerDied","Data":"b3fc4e80973e01db9e5758cb45500adc62bcd36a55b31fd982286b8067ca6187"} Nov 03 00:02:50.432573 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:50.432555 5672 scope.go:122] "RemoveContainer" containerID="b3fc4e80973e01db9e5758cb45500adc62bcd36a55b31fd982286b8067ca6187" Nov 03 00:02:50.434397 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:50.434370 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:50.434477 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:50.434438 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:50.434477 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:50.434470 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:02:51.457875 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:51.457840 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" event={"ID":"c73f440b-eeec-4ce7-861c-3171d6f6d70b","Type":"ContainerStarted","Data":"938ab0ae90dc868a61f58fa5dc0de821ea3ef9a0116de1ce6dabd94be88e345b"} Nov 03 00:02:52.212694 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:52.212659 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:02:52.213826 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:52.213794 5672 patch_prober.go:29] interesting pod/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 03 00:02:52.213972 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:52.213860 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 03 00:02:53.212397 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:53.212362 5672 patch_prober.go:29] interesting pod/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 03 00:02:53.212820 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:53.212429 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 03 00:02:54.212373 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:54.212337 5672 patch_prober.go:29] interesting pod/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 03 00:02:54.212530 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:54.212389 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 03 00:02:55.212335 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:55.212297 5672 patch_prober.go:29] interesting pod/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 03 00:02:55.212527 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:55.212350 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 03 00:02:56.021227 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:02:56.021189 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:02:56.212228 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:56.212193 5672 patch_prober.go:29] interesting pod/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 03 00:02:56.212434 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:56.212243 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 03 00:02:56.451698 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:56.451539 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:02:57.217218 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:57.217189 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:02:57.218390 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:02:57.218369 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:03:26.020676 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:03:26.020633 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:03:53.966941 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:53.966906 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4"] Nov 03 00:03:54.067785 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:54.067746 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs"] Nov 03 00:03:56.020276 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:03:56.020236 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:03:58.967480 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:58.967423 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" podUID="32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b" containerName="payload-processing" containerID="cri-o://ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd" gracePeriod=25 Nov 03 00:03:59.068441 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.068397 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" podUID="0c192e7c-1b76-4d44-9b98-05f1e3c235f7" containerName="payload-pre-processing" containerID="cri-o://a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7" gracePeriod=25 Nov 03 00:03:59.287986 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.287957 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 03 00:03:59.394884 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.394856 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-kube-api-access-vxdtk\" (UniqueName: \"kubernetes.io/projected/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-kube-api-access-vxdtk\") pod \"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b\" (UID: \"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b\") " Nov 03 00:03:59.395746 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.395058 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-plugins-config-volume\") pod \"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b\" (UID: \"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b\") " Nov 03 00:03:59.395746 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.395697 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-plugins-config-volume" pod "32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b" (UID: "32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:03:59.401897 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.401834 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-kube-api-access-vxdtk" pod "32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b" (UID: "32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b"). InnerVolumeSpecName "kube-api-access-vxdtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:03:59.414529 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.414501 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 03 00:03:59.496431 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.496402 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:03:59.496431 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.496429 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxdtk\" (UniqueName: \"kubernetes.io/projected/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b-kube-api-access-vxdtk\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:03:59.597666 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.597585 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-plugins-config-volume\") pod \"0c192e7c-1b76-4d44-9b98-05f1e3c235f7\" (UID: \"0c192e7c-1b76-4d44-9b98-05f1e3c235f7\") " Nov 03 00:03:59.597666 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.597674 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-kube-api-access-jqdbd\" (UniqueName: \"kubernetes.io/projected/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-kube-api-access-jqdbd\") pod \"0c192e7c-1b76-4d44-9b98-05f1e3c235f7\" (UID: \"0c192e7c-1b76-4d44-9b98-05f1e3c235f7\") " Nov 03 00:03:59.597978 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.597953 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-plugins-config-volume" pod "0c192e7c-1b76-4d44-9b98-05f1e3c235f7" (UID: "0c192e7c-1b76-4d44-9b98-05f1e3c235f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:03:59.599865 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.599844 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-kube-api-access-jqdbd" pod "0c192e7c-1b76-4d44-9b98-05f1e3c235f7" (UID: "0c192e7c-1b76-4d44-9b98-05f1e3c235f7"). InnerVolumeSpecName "kube-api-access-jqdbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:03:59.699135 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.699102 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:03:59.699135 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.699132 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqdbd\" (UniqueName: \"kubernetes.io/projected/0c192e7c-1b76-4d44-9b98-05f1e3c235f7-kube-api-access-jqdbd\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:03:59.923679 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.923580 5672 generic.go:356] "Generic (PLEG): container finished" podID="32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b" containerID="ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd" exitCode=0 Nov 03 00:03:59.923679 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.923639 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" Nov 03 00:03:59.923679 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.923645 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" event={"ID":"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b","Type":"ContainerDied","Data":"ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd"} Nov 03 00:03:59.923974 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.923709 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4" event={"ID":"32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b","Type":"ContainerDied","Data":"d5c4f92a4c8fb189a5c3fcc8774382701ebc80c17decf84c4f9ebfb6367e026e"} Nov 03 00:03:59.923974 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.923736 5672 scope.go:122] "RemoveContainer" containerID="ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd" Nov 03 00:03:59.925225 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.925198 5672 generic.go:356] "Generic (PLEG): container finished" podID="0c192e7c-1b76-4d44-9b98-05f1e3c235f7" containerID="a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7" exitCode=0 Nov 03 00:03:59.925344 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.925242 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" Nov 03 00:03:59.925344 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.925274 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" event={"ID":"0c192e7c-1b76-4d44-9b98-05f1e3c235f7","Type":"ContainerDied","Data":"a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7"} Nov 03 00:03:59.925344 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.925324 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs" event={"ID":"0c192e7c-1b76-4d44-9b98-05f1e3c235f7","Type":"ContainerDied","Data":"a77c8fca1c85fb85d8f8e35aec38f91b5785293fd958ed35bfc1764531ea5669"} Nov 03 00:03:59.936948 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.936932 5672 scope.go:122] "RemoveContainer" containerID="ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd" Nov 03 00:03:59.937315 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:03:59.937290 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd\": container with ID starting with ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd not found: ID does not exist" containerID="ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd" Nov 03 00:03:59.937416 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.937324 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd"} err="failed to get container status \"ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd\": rpc error: code = NotFound desc = could not find container \"ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd\": container with ID starting with ad6225a213cba1cb368707c2055293bae51269fd295c95eccee14bed08f090dd not found: ID does not exist" Nov 03 00:03:59.937416 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.937348 5672 scope.go:122] "RemoveContainer" containerID="a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7" Nov 03 00:03:59.947324 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.947299 5672 scope.go:122] "RemoveContainer" containerID="a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7" Nov 03 00:03:59.947580 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:03:59.947559 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7\": container with ID starting with a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7 not found: ID does not exist" containerID="a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7" Nov 03 00:03:59.947658 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:03:59.947589 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7"} err="failed to get container status \"a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7\": rpc error: code = NotFound desc = could not find container \"a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7\": container with ID starting with a042a479fdb15257a29ba36b4216b84093563d0c7c166264ec65a265385459f7 not found: ID does not exist" Nov 03 00:04:00.076750 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:00.076716 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs"] Nov 03 00:04:00.123235 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:00.123205 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e1be81-74488f7f8c-vr9cs"] Nov 03 00:04:00.186588 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:00.186497 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4"] Nov 03 00:04:00.191071 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:00.191038 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1be81-ccdf5dd97-9b5t4"] Nov 03 00:04:00.713660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:00.713627 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c192e7c-1b76-4d44-9b98-05f1e3c235f7" path="/var/lib/kubelet/pods/0c192e7c-1b76-4d44-9b98-05f1e3c235f7/volumes" Nov 03 00:04:00.714174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:00.714118 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b" path="/var/lib/kubelet/pods/32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b/volumes" Nov 03 00:04:13.738098 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:13.738004 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:04:24.115408 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:24.115366 5672 generic.go:356] "Generic (PLEG): container finished" podID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerID="fe62798bbd139345218ebad3f647a9afe00170ba5025926d5a6ad26ca2904e17" exitCode=0 Nov 03 00:04:24.115976 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:24.115612 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" event={"ID":"d29c3752-1e71-40ba-a9b2-bfeb439fef45","Type":"ContainerDied","Data":"fe62798bbd139345218ebad3f647a9afe00170ba5025926d5a6ad26ca2904e17"} Nov 03 00:04:24.116255 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:24.116232 5672 scope.go:122] "RemoveContainer" containerID="fe62798bbd139345218ebad3f647a9afe00170ba5025926d5a6ad26ca2904e17" Nov 03 00:04:24.120329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:24.120292 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:04:24.120460 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:24.120441 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:04:24.120698 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:24.120540 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:04:25.142174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:25.136709 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" event={"ID":"d29c3752-1e71-40ba-a9b2-bfeb439fef45","Type":"ContainerStarted","Data":"415ebfe2b6a8d9e603b11cd329b44e8cf698f9d5e2cde37bb22f8c069097c102"} Nov 03 00:04:25.608158 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:25.608126 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:04:25.608433 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:25.608400 5672 patch_prober.go:29] interesting pod/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Nov 03 00:04:25.608561 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:25.608448 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Nov 03 00:04:26.021798 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:04:26.021758 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:04:26.608386 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:26.608352 5672 patch_prober.go:29] interesting pod/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Nov 03 00:04:26.608821 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:26.608406 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Nov 03 00:04:27.608103 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:27.608065 5672 patch_prober.go:29] interesting pod/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Nov 03 00:04:27.608329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:27.608123 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Nov 03 00:04:28.608222 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:28.608180 5672 patch_prober.go:29] interesting pod/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Nov 03 00:04:28.608633 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:28.608247 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Nov 03 00:04:29.501225 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:29.501190 5672 scope.go:122] "RemoveContainer" containerID="4a4cde780674cf5dcd5e0a1490321c8e60812e1285f1f1bd2e2d960f676b8f46" Nov 03 00:04:29.608396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:29.608361 5672 patch_prober.go:29] interesting pod/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Nov 03 00:04:29.608806 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:29.608415 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Nov 03 00:04:30.613328 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:30.613296 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:04:30.613760 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:30.613717 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:04:30.614575 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:30.614557 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:04:32.045608 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:32.045575 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx"] Nov 03 00:04:32.091890 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:32.091855 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348f7197-qlljx"] Nov 03 00:04:32.714527 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:32.714494 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f24a9a-156c-4c98-b9b9-b5caeea3cee0" path="/var/lib/kubelet/pods/e7f24a9a-156c-4c98-b9b9-b5caeea3cee0/volumes" Nov 03 00:04:42.942503 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:42.942464 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49"] Nov 03 00:04:42.943022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:42.943006 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0c192e7c-1b76-4d44-9b98-05f1e3c235f7" containerName="payload-pre-processing" Nov 03 00:04:42.943088 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:42.943022 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c192e7c-1b76-4d44-9b98-05f1e3c235f7" containerName="payload-pre-processing" Nov 03 00:04:42.943088 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:42.943050 5672 cpu_manager.go:414] "containerMap: removing container" podUID="32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b" containerName="payload-processing" Nov 03 00:04:42.943088 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:42.943056 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b" containerName="payload-processing" Nov 03 00:04:42.943280 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:42.943187 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a00ebb-4fec-46ac-9ba6-4f4c7b3eb16b" containerName="payload-processing" Nov 03 00:04:42.943280 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:42.943205 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c192e7c-1b76-4d44-9b98-05f1e3c235f7" containerName="payload-pre-processing" Nov 03 00:04:42.945745 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:42.945717 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:43.041784 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.041752 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:04:43.041969 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.041793 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:04:43.041969 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.041834 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:04:43.041969 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.041857 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:04:43.044820 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.044799 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49"] Nov 03 00:04:43.082826 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.082791 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb2ps\" (UniqueName: \"kubernetes.io/projected/3a3581cf-86f2-4226-af85-47ed535b1ae0-kube-api-access-wb2ps\") pod \"maas-api-revoke-sub-90290baf-45d49\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:43.083011 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.082881 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90290baf-45d49\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:43.083011 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.082914 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90290baf-45d49\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:43.183961 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.183933 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb2ps\" (UniqueName: \"kubernetes.io/projected/3a3581cf-86f2-4226-af85-47ed535b1ae0-kube-api-access-wb2ps\") pod \"maas-api-revoke-sub-90290baf-45d49\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:43.184120 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.184005 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90290baf-45d49\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:43.184120 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.184033 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90290baf-45d49\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:43.184760 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.184739 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90290baf-45d49\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:43.186493 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.186473 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90290baf-45d49\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:43.244344 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.244311 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb2ps\" (UniqueName: \"kubernetes.io/projected/3a3581cf-86f2-4226-af85-47ed535b1ae0-kube-api-access-wb2ps\") pod \"maas-api-revoke-sub-90290baf-45d49\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:43.258800 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.258775 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:43.625562 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:43.625533 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49"] Nov 03 00:04:43.627284 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:04:43.627241 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3581cf_86f2_4226_af85_47ed535b1ae0.slice/crio-a7a10ca9f1d10806e14c8b5818040fcbb2885635563aff04ac83fbacdf3c494d WatchSource:0}: Error finding container a7a10ca9f1d10806e14c8b5818040fcbb2885635563aff04ac83fbacdf3c494d: Status 404 returned error can't find the container with id a7a10ca9f1d10806e14c8b5818040fcbb2885635563aff04ac83fbacdf3c494d Nov 03 00:04:44.297832 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:44.297785 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" event={"ID":"3a3581cf-86f2-4226-af85-47ed535b1ae0","Type":"ContainerStarted","Data":"96ac322907582eca8509bce6076bf5d8ac613f6bd7595e6d19bc56921feb7fdd"} Nov 03 00:04:44.298318 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:44.297871 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" event={"ID":"3a3581cf-86f2-4226-af85-47ed535b1ae0","Type":"ContainerStarted","Data":"a7a10ca9f1d10806e14c8b5818040fcbb2885635563aff04ac83fbacdf3c494d"} Nov 03 00:04:44.407042 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:44.406986 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" podStartSLOduration=2.406972517 podStartE2EDuration="2.406972517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:44.403643188 +0000 UTC m=+4277.810955018" watchObservedRunningTime="2026-11-03 00:04:44.406972517 +0000 UTC m=+4277.814284348" Nov 03 00:04:45.305911 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:45.305875 5672 generic.go:356] "Generic (PLEG): container finished" podID="3a3581cf-86f2-4226-af85-47ed535b1ae0" containerID="96ac322907582eca8509bce6076bf5d8ac613f6bd7595e6d19bc56921feb7fdd" exitCode=0 Nov 03 00:04:45.306322 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:45.305964 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" event={"ID":"3a3581cf-86f2-4226-af85-47ed535b1ae0","Type":"ContainerDied","Data":"96ac322907582eca8509bce6076bf5d8ac613f6bd7595e6d19bc56921feb7fdd"} Nov 03 00:04:46.484660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:46.484636 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:46.625429 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:46.625342 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-xks-ca\") pod \"3a3581cf-86f2-4226-af85-47ed535b1ae0\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " Nov 03 00:04:46.625606 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:46.625437 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3581cf-86f2-4226-af85-47ed535b1ae0-kube-api-access-wb2ps\" (UniqueName: \"kubernetes.io/projected/3a3581cf-86f2-4226-af85-47ed535b1ae0-kube-api-access-wb2ps\") pod \"3a3581cf-86f2-4226-af85-47ed535b1ae0\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " Nov 03 00:04:46.625606 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:46.625493 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-openshift-ca\") pod \"3a3581cf-86f2-4226-af85-47ed535b1ae0\" (UID: \"3a3581cf-86f2-4226-af85-47ed535b1ae0\") " Nov 03 00:04:46.626519 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:46.626486 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-openshift-ca" pod "3a3581cf-86f2-4226-af85-47ed535b1ae0" (UID: "3a3581cf-86f2-4226-af85-47ed535b1ae0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:04:46.628075 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:46.628046 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-xks-ca" pod "3a3581cf-86f2-4226-af85-47ed535b1ae0" (UID: "3a3581cf-86f2-4226-af85-47ed535b1ae0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:04:46.628272 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:46.628191 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3581cf-86f2-4226-af85-47ed535b1ae0-kube-api-access-wb2ps" pod "3a3581cf-86f2-4226-af85-47ed535b1ae0" (UID: "3a3581cf-86f2-4226-af85-47ed535b1ae0"). InnerVolumeSpecName "kube-api-access-wb2ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:04:46.727437 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:46.727411 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:46.727437 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:46.727436 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb2ps\" (UniqueName: \"kubernetes.io/projected/3a3581cf-86f2-4226-af85-47ed535b1ae0-kube-api-access-wb2ps\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:46.727619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:46.727447 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3581cf-86f2-4226-af85-47ed535b1ae0-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:04:47.322006 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:47.321973 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" event={"ID":"3a3581cf-86f2-4226-af85-47ed535b1ae0","Type":"ContainerDied","Data":"a7a10ca9f1d10806e14c8b5818040fcbb2885635563aff04ac83fbacdf3c494d"} Nov 03 00:04:47.322221 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:47.322006 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49" Nov 03 00:04:47.322221 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:47.322029 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7a10ca9f1d10806e14c8b5818040fcbb2885635563aff04ac83fbacdf3c494d" Nov 03 00:04:52.079599 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:04:52.079558 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:04:56.221804 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:04:56.221749 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:05:07.629257 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:07.629216 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl"] Nov 03 00:05:12.630238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:12.630182 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" podUID="6c95b71b-5c1a-43ec-8314-8c868fdf4a64" containerName="payload-pre-processing" containerID="cri-o://3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe" gracePeriod=25 Nov 03 00:05:12.914654 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:12.914622 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:05:12.992211 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:12.992144 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-plugins-config-volume\") pod \"6c95b71b-5c1a-43ec-8314-8c868fdf4a64\" (UID: \"6c95b71b-5c1a-43ec-8314-8c868fdf4a64\") " Nov 03 00:05:12.992436 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:12.992419 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-kube-api-access-zszsj\" (UniqueName: \"kubernetes.io/projected/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-kube-api-access-zszsj\") pod \"6c95b71b-5c1a-43ec-8314-8c868fdf4a64\" (UID: \"6c95b71b-5c1a-43ec-8314-8c868fdf4a64\") " Nov 03 00:05:12.992510 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:12.992451 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-plugins-config-volume" pod "6c95b71b-5c1a-43ec-8314-8c868fdf4a64" (UID: "6c95b71b-5c1a-43ec-8314-8c868fdf4a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:12.992860 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:12.992842 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:12.994338 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:12.994305 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-kube-api-access-zszsj" pod "6c95b71b-5c1a-43ec-8314-8c868fdf4a64" (UID: "6c95b71b-5c1a-43ec-8314-8c868fdf4a64"). InnerVolumeSpecName "kube-api-access-zszsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:13.094329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:13.094297 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zszsj\" (UniqueName: \"kubernetes.io/projected/6c95b71b-5c1a-43ec-8314-8c868fdf4a64-kube-api-access-zszsj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:13.548074 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:13.548035 5672 generic.go:356] "Generic (PLEG): container finished" podID="6c95b71b-5c1a-43ec-8314-8c868fdf4a64" containerID="3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe" exitCode=0 Nov 03 00:05:13.548320 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:13.548091 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" Nov 03 00:05:13.548320 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:13.548129 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" event={"ID":"6c95b71b-5c1a-43ec-8314-8c868fdf4a64","Type":"ContainerDied","Data":"3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe"} Nov 03 00:05:13.548320 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:13.548211 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl" event={"ID":"6c95b71b-5c1a-43ec-8314-8c868fdf4a64","Type":"ContainerDied","Data":"60b84c654787e950cc2511c0ae30dfb9190b9e5564c95c22743f3f9e2a58b931"} Nov 03 00:05:13.548320 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:13.548237 5672 scope.go:122] "RemoveContainer" containerID="3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe" Nov 03 00:05:13.560931 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:13.560904 5672 scope.go:122] "RemoveContainer" containerID="3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe" Nov 03 00:05:13.561289 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:05:13.561255 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe\": container with ID starting with 3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe not found: ID does not exist" containerID="3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe" Nov 03 00:05:13.561388 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:13.561299 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe"} err="failed to get container status \"3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe\": rpc error: code = NotFound desc = could not find container \"3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe\": container with ID starting with 3b4718a3b587461c06f98a38dc5487d5d8f94b484ebd6dc1ef234c675a3bfffe not found: ID does not exist" Nov 03 00:05:13.704630 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:13.704595 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl"] Nov 03 00:05:13.707733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:13.707708 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-666a38-64c95955ff-r24gl"] Nov 03 00:05:14.713472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:14.713437 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c95b71b-5c1a-43ec-8314-8c868fdf4a64" path="/var/lib/kubelet/pods/6c95b71b-5c1a-43ec-8314-8c868fdf4a64/volumes" Nov 03 00:05:24.372092 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:24.372057 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq"] Nov 03 00:05:26.184171 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:05:26.170640 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:05:29.373213 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.373177 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" podUID="17169a76-b5d3-4bde-b337-64ac068c8798" containerName="payload-processing" containerID="cri-o://4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f" gracePeriod=25 Nov 03 00:05:29.575971 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.575939 5672 scope.go:122] "RemoveContainer" containerID="061a6111edbd271a83560e1b3d36e5419830c79472c24c524c982af78d4d4994" Nov 03 00:05:29.645828 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.645799 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:05:29.702026 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.701997 5672 generic.go:356] "Generic (PLEG): container finished" podID="17169a76-b5d3-4bde-b337-64ac068c8798" containerID="4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f" exitCode=0 Nov 03 00:05:29.702238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.702040 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" Nov 03 00:05:29.702238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.702089 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" event={"ID":"17169a76-b5d3-4bde-b337-64ac068c8798","Type":"ContainerDied","Data":"4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f"} Nov 03 00:05:29.702238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.702175 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq" event={"ID":"17169a76-b5d3-4bde-b337-64ac068c8798","Type":"ContainerDied","Data":"459bc41e8c4c45b94c095c5de46c074841941b92b23545b01937f9039edb59f6"} Nov 03 00:05:29.702238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.702192 5672 scope.go:122] "RemoveContainer" containerID="4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f" Nov 03 00:05:29.712500 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.712484 5672 scope.go:122] "RemoveContainer" containerID="4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f" Nov 03 00:05:29.712772 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:05:29.712755 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f\": container with ID starting with 4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f not found: ID does not exist" containerID="4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f" Nov 03 00:05:29.712813 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.712781 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f"} err="failed to get container status \"4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f\": rpc error: code = NotFound desc = could not find container \"4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f\": container with ID starting with 4fd72a3e41e3b26f1193e102498d4586ef848174471348f445edf35cd333948f not found: ID does not exist" Nov 03 00:05:29.775787 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.775747 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17169a76-b5d3-4bde-b337-64ac068c8798-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17169a76-b5d3-4bde-b337-64ac068c8798-plugins-config-volume\") pod \"17169a76-b5d3-4bde-b337-64ac068c8798\" (UID: \"17169a76-b5d3-4bde-b337-64ac068c8798\") " Nov 03 00:05:29.775985 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.775875 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17169a76-b5d3-4bde-b337-64ac068c8798-kube-api-access-8gq4z\" (UniqueName: \"kubernetes.io/projected/17169a76-b5d3-4bde-b337-64ac068c8798-kube-api-access-8gq4z\") pod \"17169a76-b5d3-4bde-b337-64ac068c8798\" (UID: \"17169a76-b5d3-4bde-b337-64ac068c8798\") " Nov 03 00:05:29.776131 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.776108 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17169a76-b5d3-4bde-b337-64ac068c8798-plugins-config-volume" pod "17169a76-b5d3-4bde-b337-64ac068c8798" (UID: "17169a76-b5d3-4bde-b337-64ac068c8798"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:29.776529 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.776507 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17169a76-b5d3-4bde-b337-64ac068c8798-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:29.778225 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.778194 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17169a76-b5d3-4bde-b337-64ac068c8798-kube-api-access-8gq4z" pod "17169a76-b5d3-4bde-b337-64ac068c8798" (UID: "17169a76-b5d3-4bde-b337-64ac068c8798"). InnerVolumeSpecName "kube-api-access-8gq4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:29.877578 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:29.877496 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gq4z\" (UniqueName: \"kubernetes.io/projected/17169a76-b5d3-4bde-b337-64ac068c8798-kube-api-access-8gq4z\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:30.115449 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:30.115414 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq"] Nov 03 00:05:30.161653 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:30.161557 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9d4fc-d55b6f6d5-dtlqq"] Nov 03 00:05:30.712414 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:30.712379 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17169a76-b5d3-4bde-b337-64ac068c8798" path="/var/lib/kubelet/pods/17169a76-b5d3-4bde-b337-64ac068c8798/volumes" Nov 03 00:05:32.207008 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:32.206972 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht"] Nov 03 00:05:37.207467 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.207381 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" podUID="e73cece6-ad1e-48af-9350-c1cd2d0d05f4" containerName="payload-processing" containerID="cri-o://ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584" gracePeriod=25 Nov 03 00:05:37.474423 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.474382 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 03 00:05:37.595502 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.595455 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-kube-api-access-wn72f\" (UniqueName: \"kubernetes.io/projected/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-kube-api-access-wn72f\") pod \"e73cece6-ad1e-48af-9350-c1cd2d0d05f4\" (UID: \"e73cece6-ad1e-48af-9350-c1cd2d0d05f4\") " Nov 03 00:05:37.595713 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.595543 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-plugins-config-volume\") pod \"e73cece6-ad1e-48af-9350-c1cd2d0d05f4\" (UID: \"e73cece6-ad1e-48af-9350-c1cd2d0d05f4\") " Nov 03 00:05:37.596269 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.596235 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-plugins-config-volume" pod "e73cece6-ad1e-48af-9350-c1cd2d0d05f4" (UID: "e73cece6-ad1e-48af-9350-c1cd2d0d05f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:37.598973 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.598935 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-kube-api-access-wn72f" pod "e73cece6-ad1e-48af-9350-c1cd2d0d05f4" (UID: "e73cece6-ad1e-48af-9350-c1cd2d0d05f4"). InnerVolumeSpecName "kube-api-access-wn72f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:37.697606 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.697572 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn72f\" (UniqueName: \"kubernetes.io/projected/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-kube-api-access-wn72f\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:37.697606 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.697602 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73cece6-ad1e-48af-9350-c1cd2d0d05f4-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:05:37.758645 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.758563 5672 generic.go:356] "Generic (PLEG): container finished" podID="e73cece6-ad1e-48af-9350-c1cd2d0d05f4" containerID="ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584" exitCode=0 Nov 03 00:05:37.758645 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.758601 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" Nov 03 00:05:37.758645 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.758637 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" event={"ID":"e73cece6-ad1e-48af-9350-c1cd2d0d05f4","Type":"ContainerDied","Data":"ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584"} Nov 03 00:05:37.758905 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.758686 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht" event={"ID":"e73cece6-ad1e-48af-9350-c1cd2d0d05f4","Type":"ContainerDied","Data":"5b7b3ce6f2b536422d8ecabf7db9c1e68f572c8c4771a1c98a9ce73df3234f16"} Nov 03 00:05:37.758905 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.758711 5672 scope.go:122] "RemoveContainer" containerID="ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584" Nov 03 00:05:37.770518 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.770497 5672 scope.go:122] "RemoveContainer" containerID="ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584" Nov 03 00:05:37.770854 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:05:37.770821 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584\": container with ID starting with ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584 not found: ID does not exist" containerID="ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584" Nov 03 00:05:37.770945 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.770863 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584"} err="failed to get container status \"ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584\": rpc error: code = NotFound desc = could not find container \"ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584\": container with ID starting with ad9fc5b2e89c94752110bc7bff0c1a23d396af1344d8bd6098159b3e4f7a4584 not found: ID does not exist" Nov 03 00:05:37.911501 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.911470 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht"] Nov 03 00:05:37.915238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:37.915211 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68d89f-6748bbd99f-pnsht"] Nov 03 00:05:38.713954 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:38.713924 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73cece6-ad1e-48af-9350-c1cd2d0d05f4" path="/var/lib/kubelet/pods/e73cece6-ad1e-48af-9350-c1cd2d0d05f4/volumes" Nov 03 00:05:56.021035 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:05:56.020984 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:05:56.587160 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:05:56.587116 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:06:26.020693 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:06:26.020657 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:06:27.951332 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.951297 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5"] Nov 03 00:06:27.951842 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.951828 5672 cpu_manager.go:414] "containerMap: removing container" podUID="17169a76-b5d3-4bde-b337-64ac068c8798" containerName="payload-processing" Nov 03 00:06:27.951888 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.951842 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17169a76-b5d3-4bde-b337-64ac068c8798" containerName="payload-processing" Nov 03 00:06:27.951888 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.951854 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3a3581cf-86f2-4226-af85-47ed535b1ae0" containerName="revoke-keys" Nov 03 00:06:27.951888 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.951859 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3581cf-86f2-4226-af85-47ed535b1ae0" containerName="revoke-keys" Nov 03 00:06:27.951888 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.951872 5672 cpu_manager.go:414] "containerMap: removing container" podUID="6c95b71b-5c1a-43ec-8314-8c868fdf4a64" containerName="payload-pre-processing" Nov 03 00:06:27.951888 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.951878 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c95b71b-5c1a-43ec-8314-8c868fdf4a64" containerName="payload-pre-processing" Nov 03 00:06:27.951888 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.951886 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e73cece6-ad1e-48af-9350-c1cd2d0d05f4" containerName="payload-processing" Nov 03 00:06:27.952084 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.951891 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73cece6-ad1e-48af-9350-c1cd2d0d05f4" containerName="payload-processing" Nov 03 00:06:27.952084 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.951998 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="17169a76-b5d3-4bde-b337-64ac068c8798" containerName="payload-processing" Nov 03 00:06:27.952084 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.952006 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3581cf-86f2-4226-af85-47ed535b1ae0" containerName="revoke-keys" Nov 03 00:06:27.952084 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.952013 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c95b71b-5c1a-43ec-8314-8c868fdf4a64" containerName="payload-pre-processing" Nov 03 00:06:27.952084 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.952019 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73cece6-ad1e-48af-9350-c1cd2d0d05f4" containerName="payload-processing" Nov 03 00:06:27.954604 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:27.954579 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:28.004352 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.004325 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:06:28.050835 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.050810 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:06:28.050835 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.050831 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:06:28.051033 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.050831 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:06:28.054157 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.054125 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5"] Nov 03 00:06:28.130234 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.130205 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9688170-507e-4052-9414-846f4c01d832-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:28.130234 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.130237 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsfgb\" (UniqueName: \"kubernetes.io/projected/c9688170-507e-4052-9414-846f4c01d832-kube-api-access-jsfgb\") pod \"maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:28.130439 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.130374 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9688170-507e-4052-9414-846f4c01d832-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:28.231364 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.231329 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9688170-507e-4052-9414-846f4c01d832-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:28.231546 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.231503 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9688170-507e-4052-9414-846f4c01d832-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:28.231546 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.231525 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsfgb\" (UniqueName: \"kubernetes.io/projected/c9688170-507e-4052-9414-846f4c01d832-kube-api-access-jsfgb\") pod \"maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:28.232164 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.232125 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9688170-507e-4052-9414-846f4c01d832-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:28.233718 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.233693 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9688170-507e-4052-9414-846f4c01d832-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:28.284901 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.284876 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsfgb\" (UniqueName: \"kubernetes.io/projected/c9688170-507e-4052-9414-846f4c01d832-kube-api-access-jsfgb\") pod \"maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:28.568588 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.568513 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:28.937285 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:28.937251 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5"] Nov 03 00:06:29.114217 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:29.114184 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" event={"ID":"c9688170-507e-4052-9414-846f4c01d832","Type":"ContainerStarted","Data":"c81718430da09c859386240b53b1405c7fbfee60ddbabe93ef095030942a6481"} Nov 03 00:06:30.121615 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:30.121581 5672 generic.go:356] "Generic (PLEG): container finished" podID="c9688170-507e-4052-9414-846f4c01d832" containerID="d782a536fb69d6381e6ad0a085bfe93479d2741f7a2626d14c09c6e4fbda54fd" exitCode=0 Nov 03 00:06:30.122063 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:30.121668 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" event={"ID":"c9688170-507e-4052-9414-846f4c01d832","Type":"ContainerDied","Data":"d782a536fb69d6381e6ad0a085bfe93479d2741f7a2626d14c09c6e4fbda54fd"} Nov 03 00:06:31.296100 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:31.296075 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:31.370046 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:31.370012 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9688170-507e-4052-9414-846f4c01d832-kube-api-access-jsfgb\" (UniqueName: \"kubernetes.io/projected/c9688170-507e-4052-9414-846f4c01d832-kube-api-access-jsfgb\") pod \"c9688170-507e-4052-9414-846f4c01d832\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " Nov 03 00:06:31.370221 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:31.370194 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9688170-507e-4052-9414-846f4c01d832-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9688170-507e-4052-9414-846f4c01d832-maas-api-xks-ca\") pod \"c9688170-507e-4052-9414-846f4c01d832\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " Nov 03 00:06:31.370308 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:31.370249 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9688170-507e-4052-9414-846f4c01d832-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9688170-507e-4052-9414-846f4c01d832-maas-api-openshift-ca\") pod \"c9688170-507e-4052-9414-846f4c01d832\" (UID: \"c9688170-507e-4052-9414-846f4c01d832\") " Nov 03 00:06:31.370602 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:31.370568 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9688170-507e-4052-9414-846f4c01d832-maas-api-openshift-ca" pod "c9688170-507e-4052-9414-846f4c01d832" (UID: "c9688170-507e-4052-9414-846f4c01d832"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:31.370786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:31.370766 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9688170-507e-4052-9414-846f4c01d832-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:31.372205 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:31.372181 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9688170-507e-4052-9414-846f4c01d832-kube-api-access-jsfgb" pod "c9688170-507e-4052-9414-846f4c01d832" (UID: "c9688170-507e-4052-9414-846f4c01d832"). InnerVolumeSpecName "kube-api-access-jsfgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:31.372329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:31.372305 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9688170-507e-4052-9414-846f4c01d832-maas-api-xks-ca" pod "c9688170-507e-4052-9414-846f4c01d832" (UID: "c9688170-507e-4052-9414-846f4c01d832"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:31.475183 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:31.472371 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9688170-507e-4052-9414-846f4c01d832-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:31.475183 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:31.472407 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsfgb\" (UniqueName: \"kubernetes.io/projected/c9688170-507e-4052-9414-846f4c01d832-kube-api-access-jsfgb\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:32.143141 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:32.143111 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" Nov 03 00:06:32.143141 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:32.143114 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5" event={"ID":"c9688170-507e-4052-9414-846f4c01d832","Type":"ContainerDied","Data":"c81718430da09c859386240b53b1405c7fbfee60ddbabe93ef095030942a6481"} Nov 03 00:06:32.143447 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:32.143197 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c81718430da09c859386240b53b1405c7fbfee60ddbabe93ef095030942a6481" Nov 03 00:06:37.608504 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:37.608470 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z"] Nov 03 00:06:37.609026 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:37.608773 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" containerID="cri-o://55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856" gracePeriod=30 Nov 03 00:06:38.368338 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:38.368304 5672 patch_prober.go:29] interesting pod/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:06:38.368510 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:38.368357 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:06:38.986556 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:38.986518 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6"] Nov 03 00:06:38.987040 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:38.986769 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" containerID="cri-o://938ab0ae90dc868a61f58fa5dc0de821ea3ef9a0116de1ce6dabd94be88e345b" gracePeriod=30 Nov 03 00:06:42.219482 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.219443 5672 patch_prober.go:29] interesting pod/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:06:42.219949 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.219517 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:06:42.872812 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.872789 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 03 00:06:42.938989 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.938914 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istiod-ca-cert\") pod \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " Nov 03 00:06:42.939182 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939019 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-credential-socket\") pod \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " Nov 03 00:06:42.939182 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939110 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-token\" (UniqueName: \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-token\") pod \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " Nov 03 00:06:42.939182 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939131 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-kube-api-access-nnz4l\" (UniqueName: \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-kube-api-access-nnz4l\") pod \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " Nov 03 00:06:42.939375 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939197 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-podinfo\") pod \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " Nov 03 00:06:42.939375 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939251 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-data\") pod \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " Nov 03 00:06:42.939375 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939304 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-socket\") pod \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " Nov 03 00:06:42.939375 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939344 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-certs\") pod \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " Nov 03 00:06:42.939605 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939365 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-credential-socket" pod "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" (UID: "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:42.939605 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939383 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istiod-ca-cert" pod "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" (UID: "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:42.939605 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939411 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-envoy\") pod \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\" (UID: \"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0\") " Nov 03 00:06:42.939787 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939605 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-data" pod "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" (UID: "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:42.939787 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939658 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-socket" pod "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" (UID: "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:42.939787 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.939730 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-certs" pod "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" (UID: "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:42.940030 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.940007 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:42.940121 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.940037 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:42.940121 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.940053 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:42.940121 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.940067 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:42.940121 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.940082 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:42.941571 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.941545 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-podinfo" pod "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" (UID: "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:06:42.941691 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.941570 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-kube-api-access-nnz4l" pod "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" (UID: "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0"). InnerVolumeSpecName "kube-api-access-nnz4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:42.941933 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.941910 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-envoy" pod "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" (UID: "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:42.942026 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:42.941952 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-token" pod "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" (UID: "2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:43.041667 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.041627 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.041667 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.041655 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.041667 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.041665 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnz4l\" (UniqueName: \"kubernetes.io/projected/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-kube-api-access-nnz4l\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.041667 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.041673 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:43.256310 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.256270 5672 generic.go:356] "Generic (PLEG): container finished" podID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerID="55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856" exitCode=0 Nov 03 00:06:43.256777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.256324 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" event={"ID":"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0","Type":"ContainerDied","Data":"55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856"} Nov 03 00:06:43.256777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.256332 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" Nov 03 00:06:43.256777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.256376 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z" event={"ID":"2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0","Type":"ContainerDied","Data":"e94d51940c08d2713c232891793b9228b13b95aa2b942d906e06b8cdfcbe23e5"} Nov 03 00:06:43.256777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.256406 5672 scope.go:122] "RemoveContainer" containerID="55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856" Nov 03 00:06:43.268674 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.268651 5672 scope.go:122] "RemoveContainer" containerID="5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe" Nov 03 00:06:43.279714 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.279692 5672 scope.go:122] "RemoveContainer" containerID="55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856" Nov 03 00:06:43.279978 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:06:43.279961 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856\": container with ID starting with 55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856 not found: ID does not exist" containerID="55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856" Nov 03 00:06:43.280025 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.279985 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856"} err="failed to get container status \"55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856\": rpc error: code = NotFound desc = could not find container \"55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856\": container with ID starting with 55100dbbfda6e79311c225c8620b9ca42e6ec4627a79ba84b56d57f317067856 not found: ID does not exist" Nov 03 00:06:43.280025 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.280002 5672 scope.go:122] "RemoveContainer" containerID="5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe" Nov 03 00:06:43.280281 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:06:43.280263 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe\": container with ID starting with 5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe not found: ID does not exist" containerID="5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe" Nov 03 00:06:43.280347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.280285 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe"} err="failed to get container status \"5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe\": rpc error: code = NotFound desc = could not find container \"5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe\": container with ID starting with 5a805c59621ce56af883a008687b8f141f0cf2ef0badf549179c1badb2188dbe not found: ID does not exist" Nov 03 00:06:43.411318 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.411284 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z"] Nov 03 00:06:43.457224 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.457190 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-68d89f-openshift-default-65bb957889-hdg4z"] Nov 03 00:06:43.818942 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.818912 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw"] Nov 03 00:06:43.903865 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:43.903827 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq"] Nov 03 00:06:44.265724 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.265696 5672 generic.go:356] "Generic (PLEG): container finished" podID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerID="938ab0ae90dc868a61f58fa5dc0de821ea3ef9a0116de1ce6dabd94be88e345b" exitCode=0 Nov 03 00:06:44.266129 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.265771 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" event={"ID":"c73f440b-eeec-4ce7-861c-3171d6f6d70b","Type":"ContainerDied","Data":"938ab0ae90dc868a61f58fa5dc0de821ea3ef9a0116de1ce6dabd94be88e345b"} Nov 03 00:06:44.266129 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.265852 5672 scope.go:122] "RemoveContainer" containerID="b3fc4e80973e01db9e5758cb45500adc62bcd36a55b31fd982286b8067ca6187" Nov 03 00:06:44.294194 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:06:44.294169 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3fc4e80973e01db9e5758cb45500adc62bcd36a55b31fd982286b8067ca6187\": container with ID starting with b3fc4e80973e01db9e5758cb45500adc62bcd36a55b31fd982286b8067ca6187 not found: ID does not exist" containerID="b3fc4e80973e01db9e5758cb45500adc62bcd36a55b31fd982286b8067ca6187" Nov 03 00:06:44.296780 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.296761 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:06:44.358497 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.358417 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-data\") pod \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " Nov 03 00:06:44.358662 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.358537 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-kube-api-access-gqbzx\" (UniqueName: \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-kube-api-access-gqbzx\") pod \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " Nov 03 00:06:44.358662 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.358593 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-token\" (UniqueName: \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-token\") pod \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " Nov 03 00:06:44.358662 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.358624 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-envoy\") pod \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " Nov 03 00:06:44.358839 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.358715 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-data" pod "c73f440b-eeec-4ce7-861c-3171d6f6d70b" (UID: "c73f440b-eeec-4ce7-861c-3171d6f6d70b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:44.358839 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.358747 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-podinfo\") pod \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " Nov 03 00:06:44.358839 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.358775 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-certs\") pod \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " Nov 03 00:06:44.358839 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.358795 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-socket\") pod \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " Nov 03 00:06:44.359067 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.359035 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-socket" pod "c73f440b-eeec-4ce7-861c-3171d6f6d70b" (UID: "c73f440b-eeec-4ce7-861c-3171d6f6d70b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:44.359067 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.359046 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-credential-socket\") pod \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " Nov 03 00:06:44.359193 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.359139 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istiod-ca-cert\") pod \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\" (UID: \"c73f440b-eeec-4ce7-861c-3171d6f6d70b\") " Nov 03 00:06:44.359355 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.359327 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-credential-socket" pod "c73f440b-eeec-4ce7-861c-3171d6f6d70b" (UID: "c73f440b-eeec-4ce7-861c-3171d6f6d70b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:44.359761 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.359737 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:44.359852 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.359768 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:44.359852 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.359763 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-certs" pod "c73f440b-eeec-4ce7-861c-3171d6f6d70b" (UID: "c73f440b-eeec-4ce7-861c-3171d6f6d70b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:44.359852 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.359785 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:44.360191 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.360143 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istiod-ca-cert" pod "c73f440b-eeec-4ce7-861c-3171d6f6d70b" (UID: "c73f440b-eeec-4ce7-861c-3171d6f6d70b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:44.360973 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.360951 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-podinfo" pod "c73f440b-eeec-4ce7-861c-3171d6f6d70b" (UID: "c73f440b-eeec-4ce7-861c-3171d6f6d70b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:06:44.361094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.361076 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-envoy" pod "c73f440b-eeec-4ce7-861c-3171d6f6d70b" (UID: "c73f440b-eeec-4ce7-861c-3171d6f6d70b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:44.361269 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.361253 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-kube-api-access-gqbzx" pod "c73f440b-eeec-4ce7-861c-3171d6f6d70b" (UID: "c73f440b-eeec-4ce7-861c-3171d6f6d70b"). InnerVolumeSpecName "kube-api-access-gqbzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:44.361371 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.361355 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-token" pod "c73f440b-eeec-4ce7-861c-3171d6f6d70b" (UID: "c73f440b-eeec-4ce7-861c-3171d6f6d70b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:44.461227 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.461192 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:44.461227 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.461222 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:44.461227 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.461231 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:44.461496 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.461241 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73f440b-eeec-4ce7-861c-3171d6f6d70b-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:44.461496 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.461250 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73f440b-eeec-4ce7-861c-3171d6f6d70b-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:44.461496 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.461260 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqbzx\" (UniqueName: \"kubernetes.io/projected/c73f440b-eeec-4ce7-861c-3171d6f6d70b-kube-api-access-gqbzx\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:44.712928 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:44.712851 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" path="/var/lib/kubelet/pods/2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0/volumes" Nov 03 00:06:45.275426 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:45.275393 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" Nov 03 00:06:45.275881 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:45.275424 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6" event={"ID":"c73f440b-eeec-4ce7-861c-3171d6f6d70b","Type":"ContainerDied","Data":"fd2ca2ffd3b5a0c60f40126e45a93fb91729dfcd81b75a76b6f1731c037fa1c4"} Nov 03 00:06:45.275881 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:45.275481 5672 scope.go:122] "RemoveContainer" containerID="938ab0ae90dc868a61f58fa5dc0de821ea3ef9a0116de1ce6dabd94be88e345b" Nov 03 00:06:45.443344 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:45.442194 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6"] Nov 03 00:06:45.477430 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:45.477395 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c9d4fc-openshift-default-5db774d7-rd8m6"] Nov 03 00:06:46.713275 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:46.713242 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" path="/var/lib/kubelet/pods/c73f440b-eeec-4ce7-861c-3171d6f6d70b/volumes" Nov 03 00:06:48.431606 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.431577 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm"] Nov 03 00:06:48.432175 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432160 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" Nov 03 00:06:48.432233 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432177 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" Nov 03 00:06:48.432233 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432200 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" Nov 03 00:06:48.432233 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432206 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" Nov 03 00:06:48.432233 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432221 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c9688170-507e-4052-9414-846f4c01d832" containerName="revoke-keys" Nov 03 00:06:48.432233 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432227 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9688170-507e-4052-9414-846f4c01d832" containerName="revoke-keys" Nov 03 00:06:48.432396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432240 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" Nov 03 00:06:48.432396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432244 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" Nov 03 00:06:48.432396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432254 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" Nov 03 00:06:48.432396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432259 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" Nov 03 00:06:48.432396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432351 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" Nov 03 00:06:48.432396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432361 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" Nov 03 00:06:48.432396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432369 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9688170-507e-4052-9414-846f4c01d832" containerName="revoke-keys" Nov 03 00:06:48.432396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.432375 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e85ff01-0bf8-4f3a-92ed-ac012f54e8d0" containerName="istio-proxy" Nov 03 00:06:48.436297 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.436273 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:48.487319 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.487285 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm"] Nov 03 00:06:48.530331 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.530274 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:06:48.530331 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.530285 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:06:48.530331 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.530311 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:06:48.531655 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.531632 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:06:48.611864 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.611830 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p7xh\" (UniqueName: \"kubernetes.io/projected/a5ab99c0-962a-40c8-8b23-c4719683c3ea-kube-api-access-6p7xh\") pod \"maas-api-revoke-sub-1225456f-qzxzm\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:48.611998 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.611956 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1225456f-qzxzm\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:48.612038 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.612003 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1225456f-qzxzm\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:48.713017 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.712913 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p7xh\" (UniqueName: \"kubernetes.io/projected/a5ab99c0-962a-40c8-8b23-c4719683c3ea-kube-api-access-6p7xh\") pod \"maas-api-revoke-sub-1225456f-qzxzm\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:48.713017 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.712997 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1225456f-qzxzm\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:48.713287 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.713042 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1225456f-qzxzm\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:48.713952 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.713923 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1225456f-qzxzm\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:48.715676 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.715649 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1225456f-qzxzm\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:48.766104 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.766064 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p7xh\" (UniqueName: \"kubernetes.io/projected/a5ab99c0-962a-40c8-8b23-c4719683c3ea-kube-api-access-6p7xh\") pod \"maas-api-revoke-sub-1225456f-qzxzm\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:48.820038 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.820003 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" podUID="1c972e13-18da-4503-ad48-e54e2a31e5c9" containerName="payload-processing" containerID="cri-o://114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190" gracePeriod=25 Nov 03 00:06:48.904664 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:48.904626 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" podUID="db20f18a-6f58-4b0a-ac46-1c3a22b093dd" containerName="payload-pre-processing" containerID="cri-o://78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4" gracePeriod=25 Nov 03 00:06:49.051398 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.051355 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:49.200402 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.200371 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:06:49.234713 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.234684 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:06:49.304780 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.304690 5672 generic.go:356] "Generic (PLEG): container finished" podID="1c972e13-18da-4503-ad48-e54e2a31e5c9" containerID="114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190" exitCode=0 Nov 03 00:06:49.304780 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.304734 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" Nov 03 00:06:49.304995 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.304777 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" event={"ID":"1c972e13-18da-4503-ad48-e54e2a31e5c9","Type":"ContainerDied","Data":"114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190"} Nov 03 00:06:49.304995 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.304833 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw" event={"ID":"1c972e13-18da-4503-ad48-e54e2a31e5c9","Type":"ContainerDied","Data":"1be9afd818ea076ae700cf6bcb21b8a3af267fdf58dde2d00a60d5ed7c0e2d46"} Nov 03 00:06:49.304995 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.304858 5672 scope.go:122] "RemoveContainer" containerID="114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190" Nov 03 00:06:49.307330 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.307308 5672 generic.go:356] "Generic (PLEG): container finished" podID="db20f18a-6f58-4b0a-ac46-1c3a22b093dd" containerID="78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4" exitCode=0 Nov 03 00:06:49.307482 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.307355 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" event={"ID":"db20f18a-6f58-4b0a-ac46-1c3a22b093dd","Type":"ContainerDied","Data":"78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4"} Nov 03 00:06:49.307482 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.307367 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" Nov 03 00:06:49.307482 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.307389 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq" event={"ID":"db20f18a-6f58-4b0a-ac46-1c3a22b093dd","Type":"ContainerDied","Data":"d16f7825a80aaada260fb3609c71f30b3450f5bfa96cb5d7476cde80f4a00adc"} Nov 03 00:06:49.317971 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.317956 5672 scope.go:122] "RemoveContainer" containerID="114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190" Nov 03 00:06:49.318338 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:06:49.318312 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190\": container with ID starting with 114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190 not found: ID does not exist" containerID="114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190" Nov 03 00:06:49.318465 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.318357 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190"} err="failed to get container status \"114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190\": rpc error: code = NotFound desc = could not find container \"114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190\": container with ID starting with 114975ccf7c3aa25772b546e99b0a2f6ecca5fc33d95fac72e63c32a9ad7f190 not found: ID does not exist" Nov 03 00:06:49.318465 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.318381 5672 scope.go:122] "RemoveContainer" containerID="78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4" Nov 03 00:06:49.320724 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.320700 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-kube-api-access-l8z42\" (UniqueName: \"kubernetes.io/projected/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-kube-api-access-l8z42\") pod \"db20f18a-6f58-4b0a-ac46-1c3a22b093dd\" (UID: \"db20f18a-6f58-4b0a-ac46-1c3a22b093dd\") " Nov 03 00:06:49.320842 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.320796 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c972e13-18da-4503-ad48-e54e2a31e5c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c972e13-18da-4503-ad48-e54e2a31e5c9-plugins-config-volume\") pod \"1c972e13-18da-4503-ad48-e54e2a31e5c9\" (UID: \"1c972e13-18da-4503-ad48-e54e2a31e5c9\") " Nov 03 00:06:49.320968 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.320949 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c972e13-18da-4503-ad48-e54e2a31e5c9-kube-api-access-td254\" (UniqueName: \"kubernetes.io/projected/1c972e13-18da-4503-ad48-e54e2a31e5c9-kube-api-access-td254\") pod \"1c972e13-18da-4503-ad48-e54e2a31e5c9\" (UID: \"1c972e13-18da-4503-ad48-e54e2a31e5c9\") " Nov 03 00:06:49.321041 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.321019 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-plugins-config-volume\") pod \"db20f18a-6f58-4b0a-ac46-1c3a22b093dd\" (UID: \"db20f18a-6f58-4b0a-ac46-1c3a22b093dd\") " Nov 03 00:06:49.322105 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.321408 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c972e13-18da-4503-ad48-e54e2a31e5c9-plugins-config-volume" pod "1c972e13-18da-4503-ad48-e54e2a31e5c9" (UID: "1c972e13-18da-4503-ad48-e54e2a31e5c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:49.322105 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.321652 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-plugins-config-volume" pod "db20f18a-6f58-4b0a-ac46-1c3a22b093dd" (UID: "db20f18a-6f58-4b0a-ac46-1c3a22b093dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:49.322105 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.321753 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:49.322105 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.321774 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c972e13-18da-4503-ad48-e54e2a31e5c9-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:49.323339 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.323190 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-kube-api-access-l8z42" pod "db20f18a-6f58-4b0a-ac46-1c3a22b093dd" (UID: "db20f18a-6f58-4b0a-ac46-1c3a22b093dd"). InnerVolumeSpecName "kube-api-access-l8z42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:49.323533 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.323510 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c972e13-18da-4503-ad48-e54e2a31e5c9-kube-api-access-td254" pod "1c972e13-18da-4503-ad48-e54e2a31e5c9" (UID: "1c972e13-18da-4503-ad48-e54e2a31e5c9"). InnerVolumeSpecName "kube-api-access-td254". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:49.328449 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.328425 5672 scope.go:122] "RemoveContainer" containerID="78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4" Nov 03 00:06:49.328788 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:06:49.328759 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4\": container with ID starting with 78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4 not found: ID does not exist" containerID="78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4" Nov 03 00:06:49.328897 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.328795 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4"} err="failed to get container status \"78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4\": rpc error: code = NotFound desc = could not find container \"78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4\": container with ID starting with 78537046c0a5109e870efe78d0a666fd271895638782f12b3f21c0574a42cde4 not found: ID does not exist" Nov 03 00:06:49.423646 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.423611 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td254\" (UniqueName: \"kubernetes.io/projected/1c972e13-18da-4503-ad48-e54e2a31e5c9-kube-api-access-td254\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:49.423646 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.423648 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8z42\" (UniqueName: \"kubernetes.io/projected/db20f18a-6f58-4b0a-ac46-1c3a22b093dd-kube-api-access-l8z42\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:49.440507 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.440476 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm"] Nov 03 00:06:49.444738 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:06:49.444703 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ab99c0_962a_40c8_8b23_c4719683c3ea.slice/crio-2117a0fef2356b48ce9d4b06f1a3b9cd65359bf6de702a396eee52c78daf5c8e WatchSource:0}: Error finding container 2117a0fef2356b48ce9d4b06f1a3b9cd65359bf6de702a396eee52c78daf5c8e: Status 404 returned error can't find the container with id 2117a0fef2356b48ce9d4b06f1a3b9cd65359bf6de702a396eee52c78daf5c8e Nov 03 00:06:49.723772 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.723741 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw"] Nov 03 00:06:49.729719 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.729691 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6d61f9-5cf6747b95-8hnsw"] Nov 03 00:06:49.832477 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.832448 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq"] Nov 03 00:06:49.840054 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:49.840027 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6d61f9-5bbdc7bcbc-jzvfq"] Nov 03 00:06:50.317467 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:50.317433 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" event={"ID":"a5ab99c0-962a-40c8-8b23-c4719683c3ea","Type":"ContainerStarted","Data":"0e4ce46d896b82146abe6c525a7e2c1b9fb2916dd1664d8f73921cb77c7b44d3"} Nov 03 00:06:50.317645 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:50.317485 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" event={"ID":"a5ab99c0-962a-40c8-8b23-c4719683c3ea","Type":"ContainerStarted","Data":"2117a0fef2356b48ce9d4b06f1a3b9cd65359bf6de702a396eee52c78daf5c8e"} Nov 03 00:06:50.428264 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:50.428184 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" podStartSLOduration=2.426131028 podStartE2EDuration="2.426131028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:06:50.422740605 +0000 UTC m=+4403.830052434" watchObservedRunningTime="2026-11-03 00:06:50.426131028 +0000 UTC m=+4403.833442857" Nov 03 00:06:50.719634 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:50.719605 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c972e13-18da-4503-ad48-e54e2a31e5c9" path="/var/lib/kubelet/pods/1c972e13-18da-4503-ad48-e54e2a31e5c9/volumes" Nov 03 00:06:50.720019 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:50.720012 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db20f18a-6f58-4b0a-ac46-1c3a22b093dd" path="/var/lib/kubelet/pods/db20f18a-6f58-4b0a-ac46-1c3a22b093dd/volumes" Nov 03 00:06:51.325733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:51.325691 5672 generic.go:356] "Generic (PLEG): container finished" podID="a5ab99c0-962a-40c8-8b23-c4719683c3ea" containerID="0e4ce46d896b82146abe6c525a7e2c1b9fb2916dd1664d8f73921cb77c7b44d3" exitCode=0 Nov 03 00:06:51.325936 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:51.325775 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" event={"ID":"a5ab99c0-962a-40c8-8b23-c4719683c3ea","Type":"ContainerDied","Data":"0e4ce46d896b82146abe6c525a7e2c1b9fb2916dd1664d8f73921cb77c7b44d3"} Nov 03 00:06:52.499314 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:52.499286 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:52.661871 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:52.661785 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-openshift-ca\") pod \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " Nov 03 00:06:52.662053 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:52.661922 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ab99c0-962a-40c8-8b23-c4719683c3ea-kube-api-access-6p7xh\" (UniqueName: \"kubernetes.io/projected/a5ab99c0-962a-40c8-8b23-c4719683c3ea-kube-api-access-6p7xh\") pod \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " Nov 03 00:06:52.662053 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:52.662019 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-xks-ca\") pod \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\" (UID: \"a5ab99c0-962a-40c8-8b23-c4719683c3ea\") " Nov 03 00:06:52.662224 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:52.662194 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-openshift-ca" pod "a5ab99c0-962a-40c8-8b23-c4719683c3ea" (UID: "a5ab99c0-962a-40c8-8b23-c4719683c3ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:52.662544 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:52.662516 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:52.664058 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:52.664029 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-xks-ca" pod "a5ab99c0-962a-40c8-8b23-c4719683c3ea" (UID: "a5ab99c0-962a-40c8-8b23-c4719683c3ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:52.664182 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:52.664122 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ab99c0-962a-40c8-8b23-c4719683c3ea-kube-api-access-6p7xh" pod "a5ab99c0-962a-40c8-8b23-c4719683c3ea" (UID: "a5ab99c0-962a-40c8-8b23-c4719683c3ea"). InnerVolumeSpecName "kube-api-access-6p7xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:52.764076 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:52.764034 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ab99c0-962a-40c8-8b23-c4719683c3ea-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:52.764076 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:52.764068 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p7xh\" (UniqueName: \"kubernetes.io/projected/a5ab99c0-962a-40c8-8b23-c4719683c3ea-kube-api-access-6p7xh\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:06:53.344085 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:53.344047 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" event={"ID":"a5ab99c0-962a-40c8-8b23-c4719683c3ea","Type":"ContainerDied","Data":"2117a0fef2356b48ce9d4b06f1a3b9cd65359bf6de702a396eee52c78daf5c8e"} Nov 03 00:06:53.344284 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:53.344108 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2117a0fef2356b48ce9d4b06f1a3b9cd65359bf6de702a396eee52c78daf5c8e" Nov 03 00:06:53.344284 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:53.344056 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm" Nov 03 00:06:56.020972 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:06:56.020931 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:06:57.121005 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:06:57.120967 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:07:26.021119 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:26.021037 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:07:31.760974 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.760935 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt"] Nov 03 00:07:31.761713 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.761691 5672 cpu_manager.go:414] "containerMap: removing container" podUID="db20f18a-6f58-4b0a-ac46-1c3a22b093dd" containerName="payload-pre-processing" Nov 03 00:07:31.761813 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.761715 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db20f18a-6f58-4b0a-ac46-1c3a22b093dd" containerName="payload-pre-processing" Nov 03 00:07:31.761813 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.761755 5672 cpu_manager.go:414] "containerMap: removing container" podUID="1c972e13-18da-4503-ad48-e54e2a31e5c9" containerName="payload-processing" Nov 03 00:07:31.761813 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.761764 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c972e13-18da-4503-ad48-e54e2a31e5c9" containerName="payload-processing" Nov 03 00:07:31.761813 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.761786 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a5ab99c0-962a-40c8-8b23-c4719683c3ea" containerName="revoke-keys" Nov 03 00:07:31.761813 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.761793 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ab99c0-962a-40c8-8b23-c4719683c3ea" containerName="revoke-keys" Nov 03 00:07:31.762066 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.761935 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ab99c0-962a-40c8-8b23-c4719683c3ea" containerName="revoke-keys" Nov 03 00:07:31.762066 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.761949 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73f440b-eeec-4ce7-861c-3171d6f6d70b" containerName="istio-proxy" Nov 03 00:07:31.762066 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.761965 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="db20f18a-6f58-4b0a-ac46-1c3a22b093dd" containerName="payload-pre-processing" Nov 03 00:07:31.762066 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.761978 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c972e13-18da-4503-ad48-e54e2a31e5c9" containerName="payload-processing" Nov 03 00:07:31.766947 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.766924 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:31.805821 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.805795 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph957\" (UniqueName: \"kubernetes.io/projected/a33ce716-2859-457f-bb6e-4900373e323b-kube-api-access-ph957\") pod \"maas-api-revoke-sub-bb7d3011-k26dt\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:31.805948 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.805925 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb7d3011-k26dt\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:31.805996 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.805988 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb7d3011-k26dt\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:31.816946 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.816921 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt"] Nov 03 00:07:31.859637 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.859611 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:07:31.859637 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.859612 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:07:31.859814 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.859790 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:07:31.859894 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.859878 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:07:31.906885 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.906856 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph957\" (UniqueName: \"kubernetes.io/projected/a33ce716-2859-457f-bb6e-4900373e323b-kube-api-access-ph957\") pod \"maas-api-revoke-sub-bb7d3011-k26dt\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:31.907005 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.906963 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb7d3011-k26dt\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:31.907052 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.907033 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb7d3011-k26dt\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:31.907750 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.907731 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb7d3011-k26dt\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:31.909347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.909329 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb7d3011-k26dt\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:31.960683 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:31.960652 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph957\" (UniqueName: \"kubernetes.io/projected/a33ce716-2859-457f-bb6e-4900373e323b-kube-api-access-ph957\") pod \"maas-api-revoke-sub-bb7d3011-k26dt\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:32.079863 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:32.079783 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:32.451853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:32.451821 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt"] Nov 03 00:07:32.453191 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:07:32.453137 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33ce716_2859_457f_bb6e_4900373e323b.slice/crio-f6aabfe8c2dfd219c4df8f28920498c96e2210a160d18881a0e42c2f4757e9c0 WatchSource:0}: Error finding container f6aabfe8c2dfd219c4df8f28920498c96e2210a160d18881a0e42c2f4757e9c0: Status 404 returned error can't find the container with id f6aabfe8c2dfd219c4df8f28920498c96e2210a160d18881a0e42c2f4757e9c0 Nov 03 00:07:32.671366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:32.671327 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" event={"ID":"a33ce716-2859-457f-bb6e-4900373e323b","Type":"ContainerStarted","Data":"f6aabfe8c2dfd219c4df8f28920498c96e2210a160d18881a0e42c2f4757e9c0"} Nov 03 00:07:33.680953 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:33.680864 5672 generic.go:356] "Generic (PLEG): container finished" podID="a33ce716-2859-457f-bb6e-4900373e323b" containerID="cddfc06e59e11ad9e9a74b153820bd12e417d3e8b64990b8a108329d5d5a8f1e" exitCode=0 Nov 03 00:07:33.680953 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:33.680910 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" event={"ID":"a33ce716-2859-457f-bb6e-4900373e323b","Type":"ContainerDied","Data":"cddfc06e59e11ad9e9a74b153820bd12e417d3e8b64990b8a108329d5d5a8f1e"} Nov 03 00:07:34.826059 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:34.826030 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:34.837879 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:34.837853 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33ce716-2859-457f-bb6e-4900373e323b-kube-api-access-ph957\" (UniqueName: \"kubernetes.io/projected/a33ce716-2859-457f-bb6e-4900373e323b-kube-api-access-ph957\") pod \"a33ce716-2859-457f-bb6e-4900373e323b\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " Nov 03 00:07:34.838001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:34.837969 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-xks-ca\") pod \"a33ce716-2859-457f-bb6e-4900373e323b\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " Nov 03 00:07:34.838041 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:34.838031 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-openshift-ca\") pod \"a33ce716-2859-457f-bb6e-4900373e323b\" (UID: \"a33ce716-2859-457f-bb6e-4900373e323b\") " Nov 03 00:07:34.838425 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:34.838398 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-openshift-ca" pod "a33ce716-2859-457f-bb6e-4900373e323b" (UID: "a33ce716-2859-457f-bb6e-4900373e323b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:34.840078 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:34.840048 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-xks-ca" pod "a33ce716-2859-457f-bb6e-4900373e323b" (UID: "a33ce716-2859-457f-bb6e-4900373e323b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:07:34.840210 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:34.840079 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33ce716-2859-457f-bb6e-4900373e323b-kube-api-access-ph957" pod "a33ce716-2859-457f-bb6e-4900373e323b" (UID: "a33ce716-2859-457f-bb6e-4900373e323b"). InnerVolumeSpecName "kube-api-access-ph957". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:34.940279 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:34.940202 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:34.940279 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:34.940239 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph957\" (UniqueName: \"kubernetes.io/projected/a33ce716-2859-457f-bb6e-4900373e323b-kube-api-access-ph957\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:34.940279 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:34.940254 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33ce716-2859-457f-bb6e-4900373e323b-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:35.697798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:35.697766 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" Nov 03 00:07:35.698017 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:35.697765 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt" event={"ID":"a33ce716-2859-457f-bb6e-4900373e323b","Type":"ContainerDied","Data":"f6aabfe8c2dfd219c4df8f28920498c96e2210a160d18881a0e42c2f4757e9c0"} Nov 03 00:07:35.698017 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:35.697910 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6aabfe8c2dfd219c4df8f28920498c96e2210a160d18881a0e42c2f4757e9c0" Nov 03 00:07:42.967921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:42.967885 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5"] Nov 03 00:07:43.013282 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:43.013250 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-666a38-cxsf5"] Nov 03 00:07:43.152393 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:43.152360 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x"] Nov 03 00:07:43.152870 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:43.152838 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" podUID="671563f9-fd52-4376-9496-1bb3aa4a2254" containerName="istio-proxy" containerID="cri-o://d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d" gracePeriod=30 Nov 03 00:07:44.713424 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:44.713386 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9688170-507e-4052-9414-846f4c01d832" path="/var/lib/kubelet/pods/c9688170-507e-4052-9414-846f4c01d832/volumes" Nov 03 00:07:48.423244 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.423218 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:07:48.503894 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.503860 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/671563f9-fd52-4376-9496-1bb3aa4a2254-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671563f9-fd52-4376-9496-1bb3aa4a2254-istiod-ca-cert\") pod \"671563f9-fd52-4376-9496-1bb3aa4a2254\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " Nov 03 00:07:48.504094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.503902 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-podinfo\") pod \"671563f9-fd52-4376-9496-1bb3aa4a2254\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " Nov 03 00:07:48.504094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504025 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-credential-socket\") pod \"671563f9-fd52-4376-9496-1bb3aa4a2254\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " Nov 03 00:07:48.504094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504062 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-envoy\") pod \"671563f9-fd52-4376-9496-1bb3aa4a2254\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " Nov 03 00:07:48.504094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504087 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-kube-api-access-c79d4\" (UniqueName: \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-kube-api-access-c79d4\") pod \"671563f9-fd52-4376-9496-1bb3aa4a2254\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " Nov 03 00:07:48.504378 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504138 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-certs\") pod \"671563f9-fd52-4376-9496-1bb3aa4a2254\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " Nov 03 00:07:48.504378 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504235 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-socket\") pod \"671563f9-fd52-4376-9496-1bb3aa4a2254\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " Nov 03 00:07:48.504378 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504285 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-data\") pod \"671563f9-fd52-4376-9496-1bb3aa4a2254\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " Nov 03 00:07:48.504378 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504329 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-token\" (UniqueName: \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-token\") pod \"671563f9-fd52-4376-9496-1bb3aa4a2254\" (UID: \"671563f9-fd52-4376-9496-1bb3aa4a2254\") " Nov 03 00:07:48.504378 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504342 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671563f9-fd52-4376-9496-1bb3aa4a2254-istiod-ca-cert" pod "671563f9-fd52-4376-9496-1bb3aa4a2254" (UID: "671563f9-fd52-4376-9496-1bb3aa4a2254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:48.504661 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504435 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-certs" pod "671563f9-fd52-4376-9496-1bb3aa4a2254" (UID: "671563f9-fd52-4376-9496-1bb3aa4a2254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:48.504661 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504521 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-socket" pod "671563f9-fd52-4376-9496-1bb3aa4a2254" (UID: "671563f9-fd52-4376-9496-1bb3aa4a2254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:48.504661 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504655 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-data" pod "671563f9-fd52-4376-9496-1bb3aa4a2254" (UID: "671563f9-fd52-4376-9496-1bb3aa4a2254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:48.504841 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504370 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-credential-socket" pod "671563f9-fd52-4376-9496-1bb3aa4a2254" (UID: "671563f9-fd52-4376-9496-1bb3aa4a2254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:48.504903 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504863 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/671563f9-fd52-4376-9496-1bb3aa4a2254-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.504903 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504876 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.504903 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504885 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.504903 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504893 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.504903 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.504902 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.506455 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.506429 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-kube-api-access-c79d4" pod "671563f9-fd52-4376-9496-1bb3aa4a2254" (UID: "671563f9-fd52-4376-9496-1bb3aa4a2254"). InnerVolumeSpecName "kube-api-access-c79d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:48.506584 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.506481 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-token" pod "671563f9-fd52-4376-9496-1bb3aa4a2254" (UID: "671563f9-fd52-4376-9496-1bb3aa4a2254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:48.506653 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.506605 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-podinfo" pod "671563f9-fd52-4376-9496-1bb3aa4a2254" (UID: "671563f9-fd52-4376-9496-1bb3aa4a2254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:07:48.506653 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.506618 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-envoy" pod "671563f9-fd52-4376-9496-1bb3aa4a2254" (UID: "671563f9-fd52-4376-9496-1bb3aa4a2254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:07:48.605670 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.605638 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.605670 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.605667 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.605670 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.605677 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/671563f9-fd52-4376-9496-1bb3aa4a2254-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.605903 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.605687 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c79d4\" (UniqueName: \"kubernetes.io/projected/671563f9-fd52-4376-9496-1bb3aa4a2254-kube-api-access-c79d4\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:48.786954 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.786849 5672 generic.go:356] "Generic (PLEG): container finished" podID="671563f9-fd52-4376-9496-1bb3aa4a2254" containerID="d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d" exitCode=0 Nov 03 00:07:48.786954 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.786908 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" event={"ID":"671563f9-fd52-4376-9496-1bb3aa4a2254","Type":"ContainerDied","Data":"d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d"} Nov 03 00:07:48.786954 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.786951 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" event={"ID":"671563f9-fd52-4376-9496-1bb3aa4a2254","Type":"ContainerDied","Data":"9264b0fe5032ef12f6b6c7f0c01a959c42624719bab0ca23aae5bb4acfa99e8d"} Nov 03 00:07:48.787298 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.786912 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x" Nov 03 00:07:48.787298 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.786973 5672 scope.go:122] "RemoveContainer" containerID="d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d" Nov 03 00:07:48.797679 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.797653 5672 scope.go:122] "RemoveContainer" containerID="d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d" Nov 03 00:07:48.798011 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:48.797980 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d\": container with ID starting with d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d not found: ID does not exist" containerID="d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d" Nov 03 00:07:48.798188 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.798018 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d"} err="failed to get container status \"d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d\": rpc error: code = NotFound desc = could not find container \"d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d\": container with ID starting with d40c7db0d7e50908aa0a921b7825c1a75dce01375ef48ebac9e75678a9ceab3d not found: ID does not exist" Nov 03 00:07:48.943773 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.943736 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x"] Nov 03 00:07:48.948655 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:48.948629 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-czs9x"] Nov 03 00:07:49.104664 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.104586 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b"] Nov 03 00:07:49.105111 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.105097 5672 cpu_manager.go:414] "containerMap: removing container" podUID="671563f9-fd52-4376-9496-1bb3aa4a2254" containerName="istio-proxy" Nov 03 00:07:49.105179 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.105112 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="671563f9-fd52-4376-9496-1bb3aa4a2254" containerName="istio-proxy" Nov 03 00:07:49.105179 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.105134 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a33ce716-2859-457f-bb6e-4900373e323b" containerName="revoke-keys" Nov 03 00:07:49.105179 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.105142 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33ce716-2859-457f-bb6e-4900373e323b" containerName="revoke-keys" Nov 03 00:07:49.105295 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.105268 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33ce716-2859-457f-bb6e-4900373e323b" containerName="revoke-keys" Nov 03 00:07:49.105295 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.105284 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="671563f9-fd52-4376-9496-1bb3aa4a2254" containerName="istio-proxy" Nov 03 00:07:49.108666 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.108647 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.159143 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.159109 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-24161ec2fe36\"" Nov 03 00:07:49.159339 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.159319 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 00:07:49.159524 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.159492 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 00:07:49.159660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.159549 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-fv4qm\"" Nov 03 00:07:49.159660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.159566 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 00:07:49.159973 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.159666 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 00:07:49.163922 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.163900 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b"] Nov 03 00:07:49.211793 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.211746 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp7gx\" (UniqueName: \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-kube-api-access-kp7gx\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.211949 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.211888 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.211949 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.211947 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-routing-overlay-volume\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.212083 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.212036 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-plugins-config-volume\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.312880 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.312854 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp7gx\" (UniqueName: \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-kube-api-access-kp7gx\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.313066 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.312914 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.313066 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.312966 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-routing-overlay-volume\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.313066 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.313010 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-plugins-config-volume\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.313730 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.313704 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-routing-overlay-volume\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.313831 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.313760 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-plugins-config-volume\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.315341 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.315318 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.366035 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.365980 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp7gx\" (UniqueName: \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-kube-api-access-kp7gx\") pod \"payload-processing-external-model-58d76d4db8-gxb7b\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.421480 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.421453 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:49.831899 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:49.831875 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b"] Nov 03 00:07:49.834142 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:07:49.834111 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b1bf4a_277b_4f2d_9c9f_f0c19a56e4ad.slice/crio-e2c6a5549837256a6671327573b5a7591be819c994b74ded857dacb41de8fc7c WatchSource:0}: Error finding container e2c6a5549837256a6671327573b5a7591be819c994b74ded857dacb41de8fc7c: Status 404 returned error can't find the container with id e2c6a5549837256a6671327573b5a7591be819c994b74ded857dacb41de8fc7c Nov 03 00:07:50.713769 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:50.713738 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="671563f9-fd52-4376-9496-1bb3aa4a2254" path="/var/lib/kubelet/pods/671563f9-fd52-4376-9496-1bb3aa4a2254/volumes" Nov 03 00:07:50.801976 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:50.801938 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" event={"ID":"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad","Type":"ContainerStarted","Data":"aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75"} Nov 03 00:07:50.802186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:50.801988 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" event={"ID":"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad","Type":"ContainerStarted","Data":"e2c6a5549837256a6671327573b5a7591be819c994b74ded857dacb41de8fc7c"} Nov 03 00:07:50.802186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:50.802024 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:50.911239 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:50.911184 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" podStartSLOduration=1.911168641 podStartE2EDuration="1.911168641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:07:50.906374668 +0000 UTC m=+4464.313686492" watchObservedRunningTime="2026-11-03 00:07:50.911168641 +0000 UTC m=+4464.318480464" Nov 03 00:07:51.948788 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:51.948219 5672 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-24161ec2fe36: secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:51.948788 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:51.948250 5672 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b: secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:51.948788 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:51.948314 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials podName:46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad nodeName:}" failed. No retries permitted until 2026-11-03 00:07:52.448293934 +0000 UTC m=+4465.855605757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials") pod "payload-processing-external-model-58d76d4db8-gxb7b" (UID: "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad") : secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:52.282581 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:52.282540 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b"] Nov 03 00:07:52.454798 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:52.454759 5672 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-24161ec2fe36: secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:52.454798 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:52.454786 5672 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b: secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:52.455036 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:52.454839 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials podName:46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad nodeName:}" failed. No retries permitted until 2026-11-03 00:07:53.454825875 +0000 UTC m=+4466.862137686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials") pod "payload-processing-external-model-58d76d4db8-gxb7b" (UID: "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad") : secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:53.468962 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:53.468931 5672 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-24161ec2fe36: secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:53.468962 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:53.468957 5672 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b: secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:53.469536 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:53.469017 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials podName:46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad nodeName:}" failed. No retries permitted until 2026-11-03 00:07:55.469001351 +0000 UTC m=+4468.876313172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials") pod "payload-processing-external-model-58d76d4db8-gxb7b" (UID: "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad") : secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:55.493693 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:55.493654 5672 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-24161ec2fe36: secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:55.493693 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:55.493683 5672 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b: secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:55.494192 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:55.493737 5672 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials podName:46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad nodeName:}" failed. No retries permitted until 2026-11-03 00:07:59.493722167 +0000 UTC m=+4472.901033991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials") pod "payload-processing-external-model-58d76d4db8-gxb7b" (UID: "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad") : secret "agc-cred-24161ec2fe36" not found Nov 03 00:07:56.022469 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:56.022403 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:07:56.725087 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:56.725053 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd"] Nov 03 00:07:56.725519 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:56.725269 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" containerID="cri-o://415ebfe2b6a8d9e603b11cd329b44e8cf698f9d5e2cde37bb22f8c069097c102" gracePeriod=30 Nov 03 00:07:57.817160 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:57.817107 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" podUID="46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad" containerName="payload-processing" containerID="cri-o://aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75" gracePeriod=25 Nov 03 00:07:58.099576 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.099548 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:58.231053 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.231024 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials\" (UniqueName: \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials\") pod \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " Nov 03 00:07:58.231270 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.231066 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-routing-overlay-volume\") pod \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " Nov 03 00:07:58.231270 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.231212 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-kube-api-access-kp7gx\" (UniqueName: \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-kube-api-access-kp7gx\") pod \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " Nov 03 00:07:58.231402 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.231331 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-plugins-config-volume\") pod \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\" (UID: \"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad\") " Nov 03 00:07:58.231803 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.231683 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-routing-overlay-volume" pod "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad" (UID: "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:58.231971 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.231937 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-plugins-config-volume" pod "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad" (UID: "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:07:58.232104 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.232082 5672 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-routing-overlay-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:58.233407 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.233379 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials" pod "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad" (UID: "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:58.233658 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.233636 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-kube-api-access-kp7gx" pod "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad" (UID: "46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad"). InnerVolumeSpecName "kube-api-access-kp7gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:07:58.333655 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.333569 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:58.333655 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.333600 5672 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-provider-credentials\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:58.333655 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.333611 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp7gx\" (UniqueName: \"kubernetes.io/projected/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad-kube-api-access-kp7gx\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:07:58.887368 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.887329 5672 generic.go:356] "Generic (PLEG): container finished" podID="46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad" containerID="aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75" exitCode=0 Nov 03 00:07:58.887833 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.887376 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" Nov 03 00:07:58.887833 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.887409 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" event={"ID":"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad","Type":"ContainerDied","Data":"aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75"} Nov 03 00:07:58.887833 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.887463 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b" event={"ID":"46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad","Type":"ContainerDied","Data":"e2c6a5549837256a6671327573b5a7591be819c994b74ded857dacb41de8fc7c"} Nov 03 00:07:58.887833 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.887487 5672 scope.go:122] "RemoveContainer" containerID="aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75" Nov 03 00:07:58.897087 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.897066 5672 scope.go:122] "RemoveContainer" containerID="aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75" Nov 03 00:07:58.897372 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:07:58.897355 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75\": container with ID starting with aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75 not found: ID does not exist" containerID="aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75" Nov 03 00:07:58.897425 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:58.897380 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75"} err="failed to get container status \"aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75\": rpc error: code = NotFound desc = could not find container \"aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75\": container with ID starting with aa44a8a509721701fb570c7864289512984316a8919e01790d0018edc021ec75 not found: ID does not exist" Nov 03 00:07:59.040842 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:59.040753 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b"] Nov 03 00:07:59.043382 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:07:59.043352 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-58d76d4db8-gxb7b"] Nov 03 00:08:00.615610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:00.615571 5672 patch_prober.go:29] interesting pod/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:08:00.616056 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:00.615649 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:08:00.714033 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:00.713996 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad" path="/var/lib/kubelet/pods/46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad/volumes" Nov 03 00:08:01.915495 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:01.914831 5672 generic.go:356] "Generic (PLEG): container finished" podID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerID="415ebfe2b6a8d9e603b11cd329b44e8cf698f9d5e2cde37bb22f8c069097c102" exitCode=0 Nov 03 00:08:01.915495 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:01.915078 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" event={"ID":"d29c3752-1e71-40ba-a9b2-bfeb439fef45","Type":"ContainerDied","Data":"415ebfe2b6a8d9e603b11cd329b44e8cf698f9d5e2cde37bb22f8c069097c102"} Nov 03 00:08:01.915495 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:01.915160 5672 scope.go:122] "RemoveContainer" containerID="fe62798bbd139345218ebad3f647a9afe00170ba5025926d5a6ad26ca2904e17" Nov 03 00:08:01.995260 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:01.995199 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:08:02.077783 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.077748 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-credential-socket\") pod \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " Nov 03 00:08:02.077783 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.077788 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-data\") pod \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " Nov 03 00:08:02.078051 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.077816 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-envoy\") pod \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " Nov 03 00:08:02.078051 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.077861 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-certs\") pod \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " Nov 03 00:08:02.078051 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.077899 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istiod-ca-cert\") pod \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " Nov 03 00:08:02.078051 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.077975 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-podinfo\") pod \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " Nov 03 00:08:02.078302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078092 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-kube-api-access-h55ps\" (UniqueName: \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-kube-api-access-h55ps\") pod \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " Nov 03 00:08:02.078302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078125 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-token\" (UniqueName: \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-token\") pod \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " Nov 03 00:08:02.078302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078125 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-credential-socket" pod "d29c3752-1e71-40ba-a9b2-bfeb439fef45" (UID: "d29c3752-1e71-40ba-a9b2-bfeb439fef45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:02.078302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078136 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-data" pod "d29c3752-1e71-40ba-a9b2-bfeb439fef45" (UID: "d29c3752-1e71-40ba-a9b2-bfeb439fef45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:02.078302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078214 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-socket\") pod \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\" (UID: \"d29c3752-1e71-40ba-a9b2-bfeb439fef45\") " Nov 03 00:08:02.078647 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078347 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istiod-ca-cert" pod "d29c3752-1e71-40ba-a9b2-bfeb439fef45" (UID: "d29c3752-1e71-40ba-a9b2-bfeb439fef45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:08:02.078647 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078508 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-certs" pod "d29c3752-1e71-40ba-a9b2-bfeb439fef45" (UID: "d29c3752-1e71-40ba-a9b2-bfeb439fef45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:02.078773 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078674 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-socket" pod "d29c3752-1e71-40ba-a9b2-bfeb439fef45" (UID: "d29c3752-1e71-40ba-a9b2-bfeb439fef45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:02.078830 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078797 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:02.078830 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078815 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:02.078945 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078832 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:02.078945 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078848 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:02.078945 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.078862 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:02.080171 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.080133 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-envoy" pod "d29c3752-1e71-40ba-a9b2-bfeb439fef45" (UID: "d29c3752-1e71-40ba-a9b2-bfeb439fef45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:02.080400 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.080381 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-token" pod "d29c3752-1e71-40ba-a9b2-bfeb439fef45" (UID: "d29c3752-1e71-40ba-a9b2-bfeb439fef45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:02.080531 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.080503 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-podinfo" pod "d29c3752-1e71-40ba-a9b2-bfeb439fef45" (UID: "d29c3752-1e71-40ba-a9b2-bfeb439fef45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:08:02.080634 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.080572 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-kube-api-access-h55ps" pod "d29c3752-1e71-40ba-a9b2-bfeb439fef45" (UID: "d29c3752-1e71-40ba-a9b2-bfeb439fef45"). InnerVolumeSpecName "kube-api-access-h55ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:02.179580 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.179540 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h55ps\" (UniqueName: \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-kube-api-access-h55ps\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:02.179580 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.179573 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:02.179580 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.179583 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:02.179912 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.179593 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29c3752-1e71-40ba-a9b2-bfeb439fef45-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:02.922924 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.922890 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" Nov 03 00:08:02.923440 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.922925 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd" event={"ID":"d29c3752-1e71-40ba-a9b2-bfeb439fef45","Type":"ContainerDied","Data":"de4b8be966b05a85ec9fd0881a84022326c567e45a1e22c435d44f9166b6efab"} Nov 03 00:08:02.923440 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:02.922984 5672 scope.go:122] "RemoveContainer" containerID="415ebfe2b6a8d9e603b11cd329b44e8cf698f9d5e2cde37bb22f8c069097c102" Nov 03 00:08:03.124233 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:03.124116 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd"] Nov 03 00:08:03.171320 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:03.171283 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6d61f9-openshift-default-8485d6c9c7-xlsbd"] Nov 03 00:08:04.713430 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:04.713400 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" path="/var/lib/kubelet/pods/d29c3752-1e71-40ba-a9b2-bfeb439fef45/volumes" Nov 03 00:08:09.624607 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:09.624564 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:08:26.022073 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:08:26.022031 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:08:29.965840 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:29.965755 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn"] Nov 03 00:08:29.966363 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:29.966341 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" Nov 03 00:08:29.966424 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:29.966364 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" Nov 03 00:08:29.966424 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:29.966380 5672 cpu_manager.go:414] "containerMap: removing container" podUID="46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad" containerName="payload-processing" Nov 03 00:08:29.966424 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:29.966386 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad" containerName="payload-processing" Nov 03 00:08:29.966532 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:29.966508 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" Nov 03 00:08:29.966532 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:29.966518 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" Nov 03 00:08:29.966532 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:29.966524 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b1bf4a-277b-4f2d-9c9f-f0c19a56e4ad" containerName="payload-processing" Nov 03 00:08:29.970249 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:29.970224 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:30.019306 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.019259 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:08:30.065639 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.065607 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:08:30.065639 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.065606 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:08:30.065639 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.065609 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:08:30.068390 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.068368 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn"] Nov 03 00:08:30.108464 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.108424 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a35c6d1-h88kn\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:30.108464 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.108466 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pgxc\" (UniqueName: \"kubernetes.io/projected/626f3019-c1dc-41cd-8b2b-03310f8b2e07-kube-api-access-8pgxc\") pod \"maas-api-revoke-sub-5a35c6d1-h88kn\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:30.108701 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.108619 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a35c6d1-h88kn\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:30.209311 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.209271 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a35c6d1-h88kn\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:30.209508 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.209452 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a35c6d1-h88kn\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:30.209508 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.209489 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pgxc\" (UniqueName: \"kubernetes.io/projected/626f3019-c1dc-41cd-8b2b-03310f8b2e07-kube-api-access-8pgxc\") pod \"maas-api-revoke-sub-5a35c6d1-h88kn\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:30.210085 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.210062 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a35c6d1-h88kn\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:30.211838 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.211808 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a35c6d1-h88kn\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:30.262715 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.262676 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pgxc\" (UniqueName: \"kubernetes.io/projected/626f3019-c1dc-41cd-8b2b-03310f8b2e07-kube-api-access-8pgxc\") pod \"maas-api-revoke-sub-5a35c6d1-h88kn\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:30.283200 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.283164 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:30.651660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:30.651630 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn"] Nov 03 00:08:30.653732 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:08:30.653697 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626f3019_c1dc_41cd_8b2b_03310f8b2e07.slice/crio-46f31961b88cf7ca9a78cfaedc7ed0ffedea3613f1f99d272c53eef9dab5b46a WatchSource:0}: Error finding container 46f31961b88cf7ca9a78cfaedc7ed0ffedea3613f1f99d272c53eef9dab5b46a: Status 404 returned error can't find the container with id 46f31961b88cf7ca9a78cfaedc7ed0ffedea3613f1f99d272c53eef9dab5b46a Nov 03 00:08:31.130695 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:31.130658 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" event={"ID":"626f3019-c1dc-41cd-8b2b-03310f8b2e07","Type":"ContainerStarted","Data":"52c680365d356d3c253beb815ec6b48b34c21329d60574ae3f4736fe218ff2ab"} Nov 03 00:08:31.131181 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:31.130714 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" event={"ID":"626f3019-c1dc-41cd-8b2b-03310f8b2e07","Type":"ContainerStarted","Data":"46f31961b88cf7ca9a78cfaedc7ed0ffedea3613f1f99d272c53eef9dab5b46a"} Nov 03 00:08:31.236623 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:31.236572 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" podStartSLOduration=2.236557681 podStartE2EDuration="2.236557681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:31.236115281 +0000 UTC m=+4504.643427120" watchObservedRunningTime="2026-11-03 00:08:31.236557681 +0000 UTC m=+4504.643869511" Nov 03 00:08:32.138330 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:32.138289 5672 generic.go:356] "Generic (PLEG): container finished" podID="626f3019-c1dc-41cd-8b2b-03310f8b2e07" containerID="52c680365d356d3c253beb815ec6b48b34c21329d60574ae3f4736fe218ff2ab" exitCode=0 Nov 03 00:08:32.138767 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:32.138380 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" event={"ID":"626f3019-c1dc-41cd-8b2b-03310f8b2e07","Type":"ContainerDied","Data":"52c680365d356d3c253beb815ec6b48b34c21329d60574ae3f4736fe218ff2ab"} Nov 03 00:08:33.299101 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:33.299074 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:33.451296 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:33.451216 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-xks-ca\") pod \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " Nov 03 00:08:33.451485 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:33.451348 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-openshift-ca\") pod \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " Nov 03 00:08:33.451485 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:33.451443 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/626f3019-c1dc-41cd-8b2b-03310f8b2e07-kube-api-access-8pgxc\" (UniqueName: \"kubernetes.io/projected/626f3019-c1dc-41cd-8b2b-03310f8b2e07-kube-api-access-8pgxc\") pod \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\" (UID: \"626f3019-c1dc-41cd-8b2b-03310f8b2e07\") " Nov 03 00:08:33.451733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:33.451706 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-openshift-ca" pod "626f3019-c1dc-41cd-8b2b-03310f8b2e07" (UID: "626f3019-c1dc-41cd-8b2b-03310f8b2e07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:08:33.451888 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:33.451867 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:33.453395 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:33.453371 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-xks-ca" pod "626f3019-c1dc-41cd-8b2b-03310f8b2e07" (UID: "626f3019-c1dc-41cd-8b2b-03310f8b2e07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:08:33.453531 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:33.453514 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626f3019-c1dc-41cd-8b2b-03310f8b2e07-kube-api-access-8pgxc" pod "626f3019-c1dc-41cd-8b2b-03310f8b2e07" (UID: "626f3019-c1dc-41cd-8b2b-03310f8b2e07"). InnerVolumeSpecName "kube-api-access-8pgxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:33.552567 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:33.552534 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pgxc\" (UniqueName: \"kubernetes.io/projected/626f3019-c1dc-41cd-8b2b-03310f8b2e07-kube-api-access-8pgxc\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:33.552567 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:33.552565 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/626f3019-c1dc-41cd-8b2b-03310f8b2e07-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:08:34.152252 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:34.152219 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" Nov 03 00:08:34.152449 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:34.152253 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn" event={"ID":"626f3019-c1dc-41cd-8b2b-03310f8b2e07","Type":"ContainerDied","Data":"46f31961b88cf7ca9a78cfaedc7ed0ffedea3613f1f99d272c53eef9dab5b46a"} Nov 03 00:08:34.152449 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:08:34.152301 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46f31961b88cf7ca9a78cfaedc7ed0ffedea3613f1f99d272c53eef9dab5b46a" Nov 03 00:08:56.022211 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:08:56.022170 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:09:07.624174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.624118 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc"] Nov 03 00:09:07.624644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.624629 5672 cpu_manager.go:414] "containerMap: removing container" podUID="626f3019-c1dc-41cd-8b2b-03310f8b2e07" containerName="revoke-keys" Nov 03 00:09:07.624699 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.624645 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="626f3019-c1dc-41cd-8b2b-03310f8b2e07" containerName="revoke-keys" Nov 03 00:09:07.624699 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.624670 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" Nov 03 00:09:07.624699 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.624676 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d29c3752-1e71-40ba-a9b2-bfeb439fef45" containerName="istio-proxy" Nov 03 00:09:07.624823 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.624783 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="626f3019-c1dc-41cd-8b2b-03310f8b2e07" containerName="revoke-keys" Nov 03 00:09:07.627319 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.627298 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.723681 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.723627 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-c25940\"/\"kube-root-ca.crt\"" Nov 03 00:09:07.723937 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.723763 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-c25940\"/\"default-dockercfg-l8cwh\"" Nov 03 00:09:07.723937 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.723761 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-c25940\"/\"openshift-service-ca.crt\"" Nov 03 00:09:07.723937 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.723855 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-c25940\"/\"rate-test-model-c25940-kserve-self-signed-certs\"" Nov 03 00:09:07.726498 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.726476 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc"] Nov 03 00:09:07.753590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.753554 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0cfdca-539c-454d-bead-a56298920ea3-tls-certs\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.753772 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.753603 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-home\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.753824 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.753767 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-dshm\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.753913 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.753893 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-model-cache\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.753958 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.753927 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zgsm\" (UniqueName: \"kubernetes.io/projected/8e0cfdca-539c-454d-bead-a56298920ea3-kube-api-access-5zgsm\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.754054 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.754035 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-tmp-dir\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.855436 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.855393 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-dshm\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.855657 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.855471 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-model-cache\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.855657 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.855514 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zgsm\" (UniqueName: \"kubernetes.io/projected/8e0cfdca-539c-454d-bead-a56298920ea3-kube-api-access-5zgsm\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.855657 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.855588 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-tmp-dir\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.855836 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.855666 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0cfdca-539c-454d-bead-a56298920ea3-tls-certs\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.855836 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.855721 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-home\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.856001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.855924 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-model-cache\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.856419 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.856394 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-home\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.856731 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.856620 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-tmp-dir\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.859413 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.859387 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0cfdca-539c-454d-bead-a56298920ea3-tls-certs\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.861002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.860958 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-dshm\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.909023 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.908937 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zgsm\" (UniqueName: \"kubernetes.io/projected/8e0cfdca-539c-454d-bead-a56298920ea3-kube-api-access-5zgsm\") pod \"rate-test-model-c25940-kserve-6d646cfcc-m55sc\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:07.943157 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:07.943118 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:08.326909 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:08.326883 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc"] Nov 03 00:09:08.329007 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:09:08.328974 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e0cfdca_539c_454d_bead_a56298920ea3.slice/crio-21bd7a0584db2b3830bc0248e7f8a86b782657f3e4311d786f33281c6dfbccf0 WatchSource:0}: Error finding container 21bd7a0584db2b3830bc0248e7f8a86b782657f3e4311d786f33281c6dfbccf0: Status 404 returned error can't find the container with id 21bd7a0584db2b3830bc0248e7f8a86b782657f3e4311d786f33281c6dfbccf0 Nov 03 00:09:08.416224 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:08.416182 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" event={"ID":"8e0cfdca-539c-454d-bead-a56298920ea3","Type":"ContainerStarted","Data":"d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e"} Nov 03 00:09:08.416405 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:08.416234 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" event={"ID":"8e0cfdca-539c-454d-bead-a56298920ea3","Type":"ContainerStarted","Data":"21bd7a0584db2b3830bc0248e7f8a86b782657f3e4311d786f33281c6dfbccf0"} Nov 03 00:09:08.525401 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:08.525345 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" podStartSLOduration=1.52533099 podStartE2EDuration="1.52533099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:08.522793338 +0000 UTC m=+4541.930105202" watchObservedRunningTime="2026-11-03 00:09:08.52533099 +0000 UTC m=+4541.932642817" Nov 03 00:09:08.944524 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:08.944494 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:17.943619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:17.943585 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:17.961837 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:17.961807 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:18.500561 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:18.500530 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:09:18.925470 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:18.925393 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:09:26.020996 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:09:26.020955 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:09:47.046326 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:47.046288 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49"] Nov 03 00:09:47.092641 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:47.092608 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90290baf-45d49"] Nov 03 00:09:48.714329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:09:48.714291 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3581cf-86f2-4226-af85-47ed535b1ae0" path="/var/lib/kubelet/pods/3a3581cf-86f2-4226-af85-47ed535b1ae0/volumes" Nov 03 00:09:56.021004 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:09:56.020954 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:10:20.242466 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.242368 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8"] Nov 03 00:10:20.245929 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.245901 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:10:20.295249 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.295219 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1f668443\"" Nov 03 00:10:20.303255 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.303226 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/551678ad-8f0d-4350-a6b8-b6c48714ec9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8\" (UID: \"551678ad-8f0d-4350-a6b8-b6c48714ec9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:10:20.303409 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.303296 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xttrd\" (UniqueName: \"kubernetes.io/projected/551678ad-8f0d-4350-a6b8-b6c48714ec9c-kube-api-access-xttrd\") pod \"payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8\" (UID: \"551678ad-8f0d-4350-a6b8-b6c48714ec9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:10:20.341610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.341582 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1f668443-dockercfg-d7zhf\"" Nov 03 00:10:20.346083 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.346055 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8"] Nov 03 00:10:20.404509 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.404469 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/551678ad-8f0d-4350-a6b8-b6c48714ec9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8\" (UID: \"551678ad-8f0d-4350-a6b8-b6c48714ec9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:10:20.404697 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.404564 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttrd\" (UniqueName: \"kubernetes.io/projected/551678ad-8f0d-4350-a6b8-b6c48714ec9c-kube-api-access-xttrd\") pod \"payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8\" (UID: \"551678ad-8f0d-4350-a6b8-b6c48714ec9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:10:20.405318 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.405297 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/551678ad-8f0d-4350-a6b8-b6c48714ec9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8\" (UID: \"551678ad-8f0d-4350-a6b8-b6c48714ec9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:10:20.410904 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.410879 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr"] Nov 03 00:10:20.415112 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.415090 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:10:20.458115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.458087 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttrd\" (UniqueName: \"kubernetes.io/projected/551678ad-8f0d-4350-a6b8-b6c48714ec9c-kube-api-access-xttrd\") pod \"payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8\" (UID: \"551678ad-8f0d-4350-a6b8-b6c48714ec9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:10:20.466309 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.466286 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr"] Nov 03 00:10:20.505751 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.505587 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df42w\" (UniqueName: \"kubernetes.io/projected/1ace1b3c-df89-4751-96a3-0899c15c7d86-kube-api-access-df42w\") pod \"payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr\" (UID: \"1ace1b3c-df89-4751-96a3-0899c15c7d86\") " pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:10:20.506318 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.506125 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ace1b3c-df89-4751-96a3-0899c15c7d86-plugins-config-volume\") pod \"payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr\" (UID: \"1ace1b3c-df89-4751-96a3-0899c15c7d86\") " pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:10:20.559035 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.559003 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:10:20.607852 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.607817 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ace1b3c-df89-4751-96a3-0899c15c7d86-plugins-config-volume\") pod \"payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr\" (UID: \"1ace1b3c-df89-4751-96a3-0899c15c7d86\") " pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:10:20.608047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.607949 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df42w\" (UniqueName: \"kubernetes.io/projected/1ace1b3c-df89-4751-96a3-0899c15c7d86-kube-api-access-df42w\") pod \"payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr\" (UID: \"1ace1b3c-df89-4751-96a3-0899c15c7d86\") " pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:10:20.608704 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.608677 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ace1b3c-df89-4751-96a3-0899c15c7d86-plugins-config-volume\") pod \"payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr\" (UID: \"1ace1b3c-df89-4751-96a3-0899c15c7d86\") " pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:10:20.661762 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.661732 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df42w\" (UniqueName: \"kubernetes.io/projected/1ace1b3c-df89-4751-96a3-0899c15c7d86-kube-api-access-df42w\") pod \"payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr\" (UID: \"1ace1b3c-df89-4751-96a3-0899c15c7d86\") " pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:10:20.733913 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.733882 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:10:20.927503 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:20.927479 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8"] Nov 03 00:10:20.928739 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:10:20.928709 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551678ad_8f0d_4350_a6b8_b6c48714ec9c.slice/crio-ce42d5920b90d0aa2a7829aec1ea40e015b3fd0f855b36de1e5a305ce21e63a9 WatchSource:0}: Error finding container ce42d5920b90d0aa2a7829aec1ea40e015b3fd0f855b36de1e5a305ce21e63a9: Status 404 returned error can't find the container with id ce42d5920b90d0aa2a7829aec1ea40e015b3fd0f855b36de1e5a305ce21e63a9 Nov 03 00:10:21.101609 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:21.101585 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr"] Nov 03 00:10:21.103334 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:10:21.103301 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ace1b3c_df89_4751_96a3_0899c15c7d86.slice/crio-521a671ad2094791c0e6588c1d6b533bb0566acad082ceb41f008debbad02dd9 WatchSource:0}: Error finding container 521a671ad2094791c0e6588c1d6b533bb0566acad082ceb41f008debbad02dd9: Status 404 returned error can't find the container with id 521a671ad2094791c0e6588c1d6b533bb0566acad082ceb41f008debbad02dd9 Nov 03 00:10:21.940161 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:21.940117 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" event={"ID":"551678ad-8f0d-4350-a6b8-b6c48714ec9c","Type":"ContainerStarted","Data":"f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4"} Nov 03 00:10:21.940661 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:21.940198 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" event={"ID":"551678ad-8f0d-4350-a6b8-b6c48714ec9c","Type":"ContainerStarted","Data":"ce42d5920b90d0aa2a7829aec1ea40e015b3fd0f855b36de1e5a305ce21e63a9"} Nov 03 00:10:21.940661 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:21.940363 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:10:21.941748 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:21.941719 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" event={"ID":"1ace1b3c-df89-4751-96a3-0899c15c7d86","Type":"ContainerStarted","Data":"0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64"} Nov 03 00:10:21.941854 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:21.941758 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" event={"ID":"1ace1b3c-df89-4751-96a3-0899c15c7d86","Type":"ContainerStarted","Data":"521a671ad2094791c0e6588c1d6b533bb0566acad082ceb41f008debbad02dd9"} Nov 03 00:10:21.941854 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:21.941841 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:10:22.048635 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:22.048582 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" podStartSLOduration=2.048569593 podStartE2EDuration="2.048569593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:22.044884175 +0000 UTC m=+4615.452196004" watchObservedRunningTime="2026-11-03 00:10:22.048569593 +0000 UTC m=+4615.455881464" Nov 03 00:10:22.151963 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:22.151896 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" podStartSLOduration=2.151881393 podStartE2EDuration="2.151881393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:22.150800752 +0000 UTC m=+4615.558112581" watchObservedRunningTime="2026-11-03 00:10:22.151881393 +0000 UTC m=+4615.559193225" Nov 03 00:10:26.020599 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:10:26.020557 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:10:32.954112 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:32.954081 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:10:32.954608 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:10:32.954170 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:10:56.020964 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:10:56.020930 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:11:22.007857 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:22.007789 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:11:26.020921 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:11:26.020879 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:11:29.730579 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:29.730545 5672 scope.go:122] "RemoveContainer" containerID="96ac322907582eca8509bce6076bf5d8ac613f6bd7595e6d19bc56921feb7fdd" Nov 03 00:11:53.049493 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:53.049453 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm"] Nov 03 00:11:53.093943 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:53.093909 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1225456f-qzxzm"] Nov 03 00:11:54.719294 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:54.719261 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ab99c0-962a-40c8-8b23-c4719683c3ea" path="/var/lib/kubelet/pods/a5ab99c0-962a-40c8-8b23-c4719683c3ea/volumes" Nov 03 00:11:56.021031 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:11:56.020995 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:11:59.400817 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:59.400781 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:11:59.812001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:59.811969 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q"] Nov 03 00:11:59.816226 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:59.816203 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:11:59.868937 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:59.868902 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q"] Nov 03 00:11:59.911901 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:59.911861 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:11:59.911901 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:59.911883 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:11:59.911901 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:59.911869 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:11:59.912139 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:59.911922 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:11:59.953833 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:59.953803 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1015f300-hbz6q\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:11:59.953979 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:59.953869 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1015f300-hbz6q\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:11:59.953979 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:11:59.953946 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtgr\" (UniqueName: \"kubernetes.io/projected/d44eff2f-b580-4dbe-8ab2-0790894467f6-kube-api-access-qxtgr\") pod \"maas-api-revoke-sub-1015f300-hbz6q\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:12:00.055070 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:00.055031 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1015f300-hbz6q\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:12:00.055267 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:00.055101 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1015f300-hbz6q\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:12:00.055267 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:00.055127 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtgr\" (UniqueName: \"kubernetes.io/projected/d44eff2f-b580-4dbe-8ab2-0790894467f6-kube-api-access-qxtgr\") pod \"maas-api-revoke-sub-1015f300-hbz6q\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:12:00.055686 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:00.055662 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1015f300-hbz6q\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:12:00.057436 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:00.057416 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1015f300-hbz6q\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:12:00.108009 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:00.107946 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtgr\" (UniqueName: \"kubernetes.io/projected/d44eff2f-b580-4dbe-8ab2-0790894467f6-kube-api-access-qxtgr\") pod \"maas-api-revoke-sub-1015f300-hbz6q\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:12:00.130180 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:00.130143 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:12:00.498938 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:00.498892 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q"] Nov 03 00:12:00.500195 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:12:00.500133 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44eff2f_b580_4dbe_8ab2_0790894467f6.slice/crio-449638d3ae908020744cd516cd3bb03a4b5e9984c03bc7cd8efb74bcc4832c3e WatchSource:0}: Error finding container 449638d3ae908020744cd516cd3bb03a4b5e9984c03bc7cd8efb74bcc4832c3e: Status 404 returned error can't find the container with id 449638d3ae908020744cd516cd3bb03a4b5e9984c03bc7cd8efb74bcc4832c3e Nov 03 00:12:00.614817 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:00.614782 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" event={"ID":"d44eff2f-b580-4dbe-8ab2-0790894467f6","Type":"ContainerStarted","Data":"449638d3ae908020744cd516cd3bb03a4b5e9984c03bc7cd8efb74bcc4832c3e"} Nov 03 00:12:01.622747 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:01.622714 5672 generic.go:356] "Generic (PLEG): container finished" podID="d44eff2f-b580-4dbe-8ab2-0790894467f6" containerID="b9e607988bdfbfcc8dafc6defdd873ee8f1cfb9d6f4d55f95d367a8719c700e0" exitCode=0 Nov 03 00:12:01.623140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:01.622779 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" event={"ID":"d44eff2f-b580-4dbe-8ab2-0790894467f6","Type":"ContainerDied","Data":"b9e607988bdfbfcc8dafc6defdd873ee8f1cfb9d6f4d55f95d367a8719c700e0"} Nov 03 00:12:02.782872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:02.782842 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:12:02.889705 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:02.889627 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-openshift-ca\") pod \"d44eff2f-b580-4dbe-8ab2-0790894467f6\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " Nov 03 00:12:02.889705 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:02.889689 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44eff2f-b580-4dbe-8ab2-0790894467f6-kube-api-access-qxtgr\" (UniqueName: \"kubernetes.io/projected/d44eff2f-b580-4dbe-8ab2-0790894467f6-kube-api-access-qxtgr\") pod \"d44eff2f-b580-4dbe-8ab2-0790894467f6\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " Nov 03 00:12:02.889913 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:02.889804 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-xks-ca\") pod \"d44eff2f-b580-4dbe-8ab2-0790894467f6\" (UID: \"d44eff2f-b580-4dbe-8ab2-0790894467f6\") " Nov 03 00:12:02.890067 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:02.890037 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-openshift-ca" pod "d44eff2f-b580-4dbe-8ab2-0790894467f6" (UID: "d44eff2f-b580-4dbe-8ab2-0790894467f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:12:02.890333 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:02.890311 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:02.892442 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:02.892399 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-xks-ca" pod "d44eff2f-b580-4dbe-8ab2-0790894467f6" (UID: "d44eff2f-b580-4dbe-8ab2-0790894467f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:12:02.892577 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:02.892440 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44eff2f-b580-4dbe-8ab2-0790894467f6-kube-api-access-qxtgr" pod "d44eff2f-b580-4dbe-8ab2-0790894467f6" (UID: "d44eff2f-b580-4dbe-8ab2-0790894467f6"). InnerVolumeSpecName "kube-api-access-qxtgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:12:02.991644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:02.991608 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxtgr\" (UniqueName: \"kubernetes.io/projected/d44eff2f-b580-4dbe-8ab2-0790894467f6-kube-api-access-qxtgr\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:02.991644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:02.991637 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44eff2f-b580-4dbe-8ab2-0790894467f6-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:03.636451 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:03.636412 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" event={"ID":"d44eff2f-b580-4dbe-8ab2-0790894467f6","Type":"ContainerDied","Data":"449638d3ae908020744cd516cd3bb03a4b5e9984c03bc7cd8efb74bcc4832c3e"} Nov 03 00:12:03.636642 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:03.636466 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="449638d3ae908020744cd516cd3bb03a4b5e9984c03bc7cd8efb74bcc4832c3e" Nov 03 00:12:03.636642 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:03.636428 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q" Nov 03 00:12:04.221901 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:04.221862 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp"] Nov 03 00:12:04.232353 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:04.232324 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z"] Nov 03 00:12:16.077724 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.077682 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s"] Nov 03 00:12:16.078486 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.078465 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d44eff2f-b580-4dbe-8ab2-0790894467f6" containerName="revoke-keys" Nov 03 00:12:16.078589 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.078487 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44eff2f-b580-4dbe-8ab2-0790894467f6" containerName="revoke-keys" Nov 03 00:12:16.078698 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.078683 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44eff2f-b580-4dbe-8ab2-0790894467f6" containerName="revoke-keys" Nov 03 00:12:16.082202 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.082175 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:16.131755 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.131708 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:12:16.135392 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.135349 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5825a40-tnq7s\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:16.135765 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.135738 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5825a40-tnq7s\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:16.135924 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.135855 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xrjn\" (UniqueName: \"kubernetes.io/projected/14742ee8-8770-4a65-98b7-abdb9d054251-kube-api-access-6xrjn\") pod \"maas-api-revoke-sub-e5825a40-tnq7s\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:16.179204 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.179140 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:12:16.179204 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.179176 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:12:16.179432 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.179277 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:12:16.182054 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.182035 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s"] Nov 03 00:12:16.236910 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.236851 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5825a40-tnq7s\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:16.237344 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.237008 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5825a40-tnq7s\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:16.237344 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.237078 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xrjn\" (UniqueName: \"kubernetes.io/projected/14742ee8-8770-4a65-98b7-abdb9d054251-kube-api-access-6xrjn\") pod \"maas-api-revoke-sub-e5825a40-tnq7s\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:16.237890 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.237869 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5825a40-tnq7s\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:16.240085 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.240060 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5825a40-tnq7s\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:16.293084 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.293047 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xrjn\" (UniqueName: \"kubernetes.io/projected/14742ee8-8770-4a65-98b7-abdb9d054251-kube-api-access-6xrjn\") pod \"maas-api-revoke-sub-e5825a40-tnq7s\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:16.395456 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.395370 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:16.774168 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:16.774113 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s"] Nov 03 00:12:16.776104 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:12:16.776068 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14742ee8_8770_4a65_98b7_abdb9d054251.slice/crio-bfec7b176e847138ec170a64fee6281460ed132de1435cf7f6c4611549c7df8b WatchSource:0}: Error finding container bfec7b176e847138ec170a64fee6281460ed132de1435cf7f6c4611549c7df8b: Status 404 returned error can't find the container with id bfec7b176e847138ec170a64fee6281460ed132de1435cf7f6c4611549c7df8b Nov 03 00:12:17.733391 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:17.733356 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" event={"ID":"14742ee8-8770-4a65-98b7-abdb9d054251","Type":"ContainerStarted","Data":"cba68e4c4bdda5b3690349169705c58e15b57416e2f58387ddf564eeed3cb55d"} Nov 03 00:12:17.733912 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:17.733413 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" event={"ID":"14742ee8-8770-4a65-98b7-abdb9d054251","Type":"ContainerStarted","Data":"bfec7b176e847138ec170a64fee6281460ed132de1435cf7f6c4611549c7df8b"} Nov 03 00:12:17.843350 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:17.843297 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" podStartSLOduration=1.8432825369999999 podStartE2EDuration="1.843282537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:17.841400132 +0000 UTC m=+4731.248711963" watchObservedRunningTime="2026-11-03 00:12:17.843282537 +0000 UTC m=+4731.250594407" Nov 03 00:12:18.741497 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:18.741458 5672 generic.go:356] "Generic (PLEG): container finished" podID="14742ee8-8770-4a65-98b7-abdb9d054251" containerID="cba68e4c4bdda5b3690349169705c58e15b57416e2f58387ddf564eeed3cb55d" exitCode=0 Nov 03 00:12:18.741894 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:18.741700 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" event={"ID":"14742ee8-8770-4a65-98b7-abdb9d054251","Type":"ContainerDied","Data":"cba68e4c4bdda5b3690349169705c58e15b57416e2f58387ddf564eeed3cb55d"} Nov 03 00:12:19.243433 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.243388 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" containerID="cri-o://d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62" gracePeriod=45 Nov 03 00:12:19.252619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.252581 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" containerID="cri-o://cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d" gracePeriod=45 Nov 03 00:12:19.403290 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.402910 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": dial tcp 10.134.0.44:8000: connect: connection refused" Nov 03 00:12:19.582106 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.582077 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 03 00:12:19.586782 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.586759 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 03 00:12:19.681809 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.681771 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-model-cache\") pod \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " Nov 03 00:12:19.682010 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.681866 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-tmp-dir\") pod \"13f2410f-6859-45b8-8f8b-2397e063ba1e\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " Nov 03 00:12:19.682010 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.681898 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tmp-dir\") pod \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " Nov 03 00:12:19.682010 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.681937 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f2410f-6859-45b8-8f8b-2397e063ba1e-kube-api-access-hgjhp\" (UniqueName: \"kubernetes.io/projected/13f2410f-6859-45b8-8f8b-2397e063ba1e-kube-api-access-hgjhp\") pod \"13f2410f-6859-45b8-8f8b-2397e063ba1e\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " Nov 03 00:12:19.682010 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.681976 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tls-certs\") pod \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " Nov 03 00:12:19.682010 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682006 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13f2410f-6859-45b8-8f8b-2397e063ba1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/13f2410f-6859-45b8-8f8b-2397e063ba1e-tls-certs\") pod \"13f2410f-6859-45b8-8f8b-2397e063ba1e\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " Nov 03 00:12:19.682366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682044 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-dshm\") pod \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " Nov 03 00:12:19.682366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682072 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-model-cache\") pod \"13f2410f-6859-45b8-8f8b-2397e063ba1e\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " Nov 03 00:12:19.682366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682101 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-model-cache" pod "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" (UID: "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:19.682366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682128 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-dshm\") pod \"13f2410f-6859-45b8-8f8b-2397e063ba1e\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " Nov 03 00:12:19.682366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682201 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-home\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-home\") pod \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " Nov 03 00:12:19.682366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682231 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-kube-api-access-lkn8g\" (UniqueName: \"kubernetes.io/projected/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-kube-api-access-lkn8g\") pod \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\" (UID: \"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78\") " Nov 03 00:12:19.682366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682334 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-home\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-home\") pod \"13f2410f-6859-45b8-8f8b-2397e063ba1e\" (UID: \"13f2410f-6859-45b8-8f8b-2397e063ba1e\") " Nov 03 00:12:19.682366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682333 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tmp-dir" pod "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" (UID: "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:19.682820 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682535 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-tmp-dir" pod "13f2410f-6859-45b8-8f8b-2397e063ba1e" (UID: "13f2410f-6859-45b8-8f8b-2397e063ba1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:19.682820 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682730 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-home" pod "13f2410f-6859-45b8-8f8b-2397e063ba1e" (UID: "13f2410f-6859-45b8-8f8b-2397e063ba1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:19.682938 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682915 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.682938 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682923 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-model-cache" pod "13f2410f-6859-45b8-8f8b-2397e063ba1e" (UID: "13f2410f-6859-45b8-8f8b-2397e063ba1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:19.683067 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682962 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.683067 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682979 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.683067 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.682991 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.683398 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.683374 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-home" pod "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" (UID: "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:19.688946 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.688923 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tls-certs" pod "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" (UID: "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:12:19.691543 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.691512 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-dshm" pod "13f2410f-6859-45b8-8f8b-2397e063ba1e" (UID: "13f2410f-6859-45b8-8f8b-2397e063ba1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:19.691654 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.691639 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-dshm" pod "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" (UID: "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:12:19.691732 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.691718 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f2410f-6859-45b8-8f8b-2397e063ba1e-tls-certs" pod "13f2410f-6859-45b8-8f8b-2397e063ba1e" (UID: "13f2410f-6859-45b8-8f8b-2397e063ba1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:12:19.691831 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.691815 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f2410f-6859-45b8-8f8b-2397e063ba1e-kube-api-access-hgjhp" pod "13f2410f-6859-45b8-8f8b-2397e063ba1e" (UID: "13f2410f-6859-45b8-8f8b-2397e063ba1e"). InnerVolumeSpecName "kube-api-access-hgjhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:12:19.693539 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.693512 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-kube-api-access-lkn8g" pod "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" (UID: "1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78"). InnerVolumeSpecName "kube-api-access-lkn8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:12:19.763037 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.762997 5672 generic.go:356] "Generic (PLEG): container finished" podID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerID="d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62" exitCode=0 Nov 03 00:12:19.763601 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.763053 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" Nov 03 00:12:19.763601 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.763079 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" event={"ID":"13f2410f-6859-45b8-8f8b-2397e063ba1e","Type":"ContainerDied","Data":"d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62"} Nov 03 00:12:19.763601 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.763129 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp" event={"ID":"13f2410f-6859-45b8-8f8b-2397e063ba1e","Type":"ContainerDied","Data":"562769ad925ffa675e5575b6fcc7a2fa7a59a377a167f5104aae7cd91b275c86"} Nov 03 00:12:19.763601 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.763179 5672 scope.go:122] "RemoveContainer" containerID="d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62" Nov 03 00:12:19.766712 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.766630 5672 generic.go:356] "Generic (PLEG): container finished" podID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerID="cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d" exitCode=0 Nov 03 00:12:19.767074 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.767049 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" Nov 03 00:12:19.773048 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.773024 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" event={"ID":"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78","Type":"ContainerDied","Data":"cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d"} Nov 03 00:12:19.773191 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.773064 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z" event={"ID":"1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78","Type":"ContainerDied","Data":"f9a5b567fd8cd6fb7e329468d7ae99ec3897363ba103ece384a8c8f77b44c60a"} Nov 03 00:12:19.785343 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.785110 5672 scope.go:122] "RemoveContainer" containerID="d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62" Nov 03 00:12:19.786094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.785711 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgjhp\" (UniqueName: \"kubernetes.io/projected/13f2410f-6859-45b8-8f8b-2397e063ba1e-kube-api-access-hgjhp\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.786094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.785736 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.786094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.785746 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f2410f-6859-45b8-8f8b-2397e063ba1e-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.786094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.785754 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.786094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.785762 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.786094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.785772 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f2410f-6859-45b8-8f8b-2397e063ba1e-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.786094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.785780 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.786094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.785788 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkn8g\" (UniqueName: \"kubernetes.io/projected/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78-kube-api-access-lkn8g\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:19.795660 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:12:19.795442 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62\": container with ID starting with d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62 not found: ID does not exist" containerID="d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62" Nov 03 00:12:19.795660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.795485 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62"} err="failed to get container status \"d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62\": rpc error: code = NotFound desc = could not find container \"d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62\": container with ID starting with d4dcb7ee407b368e8619caaa031be50fad403e2485184bf2d933196ab3866e62 not found: ID does not exist" Nov 03 00:12:19.795660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.795511 5672 scope.go:122] "RemoveContainer" containerID="cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d" Nov 03 00:12:19.821454 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.820626 5672 scope.go:122] "RemoveContainer" containerID="cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d" Nov 03 00:12:19.821544 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:12:19.821518 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d\": container with ID starting with cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d not found: ID does not exist" containerID="cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d" Nov 03 00:12:19.821610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.821559 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d"} err="failed to get container status \"cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d\": rpc error: code = NotFound desc = could not find container \"cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d\": container with ID starting with cdc7b75523c8b1f0cd9c8cf873826e9b0ffeb0e679cc9aa9097e7609de55291d not found: ID does not exist" Nov 03 00:12:19.951219 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.951130 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp"] Nov 03 00:12:19.955410 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.955377 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/premium-w0-59f97a-kserve-7554cc7c55-4jbjp"] Nov 03 00:12:19.959849 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:19.959829 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:20.057138 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.057057 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z"] Nov 03 00:12:20.061240 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.061213 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/simulator-w0-59f97a-kserve-74fcd7d5b6-lln7z"] Nov 03 00:12:20.089396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.089368 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-xks-ca\") pod \"14742ee8-8770-4a65-98b7-abdb9d054251\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " Nov 03 00:12:20.089505 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.089421 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14742ee8-8770-4a65-98b7-abdb9d054251-kube-api-access-6xrjn\" (UniqueName: \"kubernetes.io/projected/14742ee8-8770-4a65-98b7-abdb9d054251-kube-api-access-6xrjn\") pod \"14742ee8-8770-4a65-98b7-abdb9d054251\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " Nov 03 00:12:20.089505 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.089502 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-openshift-ca\") pod \"14742ee8-8770-4a65-98b7-abdb9d054251\" (UID: \"14742ee8-8770-4a65-98b7-abdb9d054251\") " Nov 03 00:12:20.089906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.089873 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-openshift-ca" pod "14742ee8-8770-4a65-98b7-abdb9d054251" (UID: "14742ee8-8770-4a65-98b7-abdb9d054251"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:12:20.091461 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.091437 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-xks-ca" pod "14742ee8-8770-4a65-98b7-abdb9d054251" (UID: "14742ee8-8770-4a65-98b7-abdb9d054251"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:12:20.091567 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.091548 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14742ee8-8770-4a65-98b7-abdb9d054251-kube-api-access-6xrjn" pod "14742ee8-8770-4a65-98b7-abdb9d054251" (UID: "14742ee8-8770-4a65-98b7-abdb9d054251"). InnerVolumeSpecName "kube-api-access-6xrjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:12:20.191030 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.190995 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:20.191030 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.191026 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xrjn\" (UniqueName: \"kubernetes.io/projected/14742ee8-8770-4a65-98b7-abdb9d054251-kube-api-access-6xrjn\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:20.191030 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.191036 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14742ee8-8770-4a65-98b7-abdb9d054251-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:12:20.713514 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.713477 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" path="/var/lib/kubelet/pods/13f2410f-6859-45b8-8f8b-2397e063ba1e/volumes" Nov 03 00:12:20.713927 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.713913 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" path="/var/lib/kubelet/pods/1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78/volumes" Nov 03 00:12:20.778591 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.778554 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" event={"ID":"14742ee8-8770-4a65-98b7-abdb9d054251","Type":"ContainerDied","Data":"bfec7b176e847138ec170a64fee6281460ed132de1435cf7f6c4611549c7df8b"} Nov 03 00:12:20.779057 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.778609 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfec7b176e847138ec170a64fee6281460ed132de1435cf7f6c4611549c7df8b" Nov 03 00:12:20.779057 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:20.778621 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s" Nov 03 00:12:26.021516 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:12:26.021470 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:12:29.756291 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:29.756255 5672 scope.go:122] "RemoveContainer" containerID="d782a536fb69d6381e6ad0a085bfe93479d2741f7a2626d14c09c6e4fbda54fd" Nov 03 00:12:35.043018 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:35.042984 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt"] Nov 03 00:12:35.089031 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:35.088999 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb7d3011-k26dt"] Nov 03 00:12:36.712574 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:36.712540 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33ce716-2859-457f-bb6e-4900373e323b" path="/var/lib/kubelet/pods/a33ce716-2859-457f-bb6e-4900373e323b/volumes" Nov 03 00:12:50.161054 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.161010 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb"] Nov 03 00:12:50.161822 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.161799 5672 cpu_manager.go:414] "containerMap: removing container" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" Nov 03 00:12:50.161932 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.161823 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" Nov 03 00:12:50.161932 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.161869 5672 cpu_manager.go:414] "containerMap: removing container" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" Nov 03 00:12:50.161932 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.161879 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" Nov 03 00:12:50.161932 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.161902 5672 cpu_manager.go:414] "containerMap: removing container" podUID="14742ee8-8770-4a65-98b7-abdb9d054251" containerName="revoke-keys" Nov 03 00:12:50.161932 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.161910 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14742ee8-8770-4a65-98b7-abdb9d054251" containerName="revoke-keys" Nov 03 00:12:50.162242 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.162037 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f2410f-6859-45b8-8f8b-2397e063ba1e" containerName="main" Nov 03 00:12:50.162242 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.162051 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="14742ee8-8770-4a65-98b7-abdb9d054251" containerName="revoke-keys" Nov 03 00:12:50.162242 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.162067 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9b5e4a-b530-4bf9-a6a0-c0bf99af8d78" containerName="main" Nov 03 00:12:50.166029 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.166002 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.214505 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.214474 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-59f97a-openshift-default-dockercfg-r7nkq\"" Nov 03 00:12:50.238019 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.237984 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.238019 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.238019 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.238281 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.238036 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.238281 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.238226 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbx4z\" (UniqueName: \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-kube-api-access-nbx4z\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.238411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.238294 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.238411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.238353 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff877c1-a472-40e6-9253-98ce9a9d2098-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.238519 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.238493 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.238577 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.238523 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.238577 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.238551 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.264904 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.264872 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb"] Nov 03 00:12:50.339991 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.339959 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340211 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340004 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340211 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340039 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340211 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340169 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340211 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340202 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340456 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340276 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbx4z\" (UniqueName: \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-kube-api-access-nbx4z\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340456 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340345 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340456 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340408 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff877c1-a472-40e6-9253-98ce9a9d2098-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340465 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340547 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340751 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340719 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340843 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340822 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.340919 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.340898 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.341140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.341118 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff877c1-a472-40e6-9253-98ce9a9d2098-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.342919 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.342894 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.343017 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.342949 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.392669 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.392641 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.392852 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.392828 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbx4z\" (UniqueName: \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-kube-api-access-nbx4z\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.488917 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.488888 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:50.913060 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.912980 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb"] Nov 03 00:12:50.950709 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:12:50.950675 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff877c1_a472_40e6_9253_98ce9a9d2098.slice/crio-2727608d696759bfbc682802ef0393bb3d8ae454fb18323b1343b52ea2336253 WatchSource:0}: Error finding container 2727608d696759bfbc682802ef0393bb3d8ae454fb18323b1343b52ea2336253: Status 404 returned error can't find the container with id 2727608d696759bfbc682802ef0393bb3d8ae454fb18323b1343b52ea2336253 Nov 03 00:12:50.953572 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.953535 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:12:50.953692 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.953619 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:12:50.953692 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.953662 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:12:50.992184 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:50.992135 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" event={"ID":"0ff877c1-a472-40e6-9253-98ce9a9d2098","Type":"ContainerStarted","Data":"2727608d696759bfbc682802ef0393bb3d8ae454fb18323b1343b52ea2336253"} Nov 03 00:12:52.000062 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:52.000022 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" event={"ID":"0ff877c1-a472-40e6-9253-98ce9a9d2098","Type":"ContainerStarted","Data":"20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c"} Nov 03 00:12:52.158517 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:52.158446 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" podStartSLOduration=2.158431673 podStartE2EDuration="2.158431673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:52.157418273 +0000 UTC m=+4765.564730103" watchObservedRunningTime="2026-11-03 00:12:52.158431673 +0000 UTC m=+4765.565743503" Nov 03 00:12:52.489775 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:52.489745 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:12:52.491233 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:52.491206 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Nov 03 00:12:52.491390 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:52.491260 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" podUID="0ff877c1-a472-40e6-9253-98ce9a9d2098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Nov 03 00:12:53.489630 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:53.489595 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Nov 03 00:12:53.490132 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:53.489652 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" podUID="0ff877c1-a472-40e6-9253-98ce9a9d2098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Nov 03 00:12:54.123414 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:54.123376 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb"] Nov 03 00:12:54.489893 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:54.489859 5672 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body= Nov 03 00:12:54.490403 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:54.489916 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" podUID="0ff877c1-a472-40e6-9253-98ce9a9d2098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" Nov 03 00:12:54.563048 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:54.563010 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:12:55.023251 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:55.023210 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" podUID="0ff877c1-a472-40e6-9253-98ce9a9d2098" containerName="istio-proxy" containerID="cri-o://20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c" gracePeriod=30 Nov 03 00:12:56.021709 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:12:56.021671 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:12:59.384817 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.384777 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr"] Nov 03 00:12:59.389504 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.389484 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:12:59.441984 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.441955 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr"] Nov 03 00:12:59.484639 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.484609 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:12:59.484639 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.484631 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:12:59.484814 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.484667 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:12:59.484814 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.484785 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:12:59.575452 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.575424 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8gmj\" (UniqueName: \"kubernetes.io/projected/e28074d5-66da-451a-9bc0-fd947c8c9942-kube-api-access-q8gmj\") pod \"maas-api-revoke-sub-2b0cc116-b6jfr\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:12:59.575623 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.575514 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b0cc116-b6jfr\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:12:59.575667 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.575635 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b0cc116-b6jfr\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:12:59.677516 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.677436 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b0cc116-b6jfr\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:12:59.677678 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.677518 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b0cc116-b6jfr\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:12:59.677678 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.677649 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8gmj\" (UniqueName: \"kubernetes.io/projected/e28074d5-66da-451a-9bc0-fd947c8c9942-kube-api-access-q8gmj\") pod \"maas-api-revoke-sub-2b0cc116-b6jfr\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:12:59.678329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.678301 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b0cc116-b6jfr\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:12:59.679863 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.679844 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b0cc116-b6jfr\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:12:59.730589 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:12:59.730554 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8gmj\" (UniqueName: \"kubernetes.io/projected/e28074d5-66da-451a-9bc0-fd947c8c9942-kube-api-access-q8gmj\") pod \"maas-api-revoke-sub-2b0cc116-b6jfr\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:13:00.003327 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:00.003298 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:13:00.368866 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:00.368827 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr"] Nov 03 00:13:00.370103 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:13:00.370029 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28074d5_66da_451a_9bc0_fd947c8c9942.slice/crio-4299bb615fad8e2ee8e8a844e0b7996fb6462f9d0437fef60dd8a3f4346de24d WatchSource:0}: Error finding container 4299bb615fad8e2ee8e8a844e0b7996fb6462f9d0437fef60dd8a3f4346de24d: Status 404 returned error can't find the container with id 4299bb615fad8e2ee8e8a844e0b7996fb6462f9d0437fef60dd8a3f4346de24d Nov 03 00:13:01.091093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:01.091053 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" event={"ID":"e28074d5-66da-451a-9bc0-fd947c8c9942","Type":"ContainerStarted","Data":"7f6331bfaf730ed215a3cd77fc461add590b29d18030e954950060193d6c8a8d"} Nov 03 00:13:01.091602 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:01.091116 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" event={"ID":"e28074d5-66da-451a-9bc0-fd947c8c9942","Type":"ContainerStarted","Data":"4299bb615fad8e2ee8e8a844e0b7996fb6462f9d0437fef60dd8a3f4346de24d"} Nov 03 00:13:01.200673 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:01.200626 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" podStartSLOduration=2.200609609 podStartE2EDuration="2.200609609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:01.197590651 +0000 UTC m=+4774.604902484" watchObservedRunningTime="2026-11-03 00:13:01.200609609 +0000 UTC m=+4774.607921447" Nov 03 00:13:02.100223 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:02.100187 5672 generic.go:356] "Generic (PLEG): container finished" podID="e28074d5-66da-451a-9bc0-fd947c8c9942" containerID="7f6331bfaf730ed215a3cd77fc461add590b29d18030e954950060193d6c8a8d" exitCode=0 Nov 03 00:13:02.100223 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:02.100222 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" event={"ID":"e28074d5-66da-451a-9bc0-fd947c8c9942","Type":"ContainerDied","Data":"7f6331bfaf730ed215a3cd77fc461add590b29d18030e954950060193d6c8a8d"} Nov 03 00:13:03.258472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.258442 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:13:03.310134 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.310099 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-xks-ca\") pod \"e28074d5-66da-451a-9bc0-fd947c8c9942\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " Nov 03 00:13:03.310356 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.310144 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28074d5-66da-451a-9bc0-fd947c8c9942-kube-api-access-q8gmj\" (UniqueName: \"kubernetes.io/projected/e28074d5-66da-451a-9bc0-fd947c8c9942-kube-api-access-q8gmj\") pod \"e28074d5-66da-451a-9bc0-fd947c8c9942\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " Nov 03 00:13:03.310356 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.310195 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-openshift-ca\") pod \"e28074d5-66da-451a-9bc0-fd947c8c9942\" (UID: \"e28074d5-66da-451a-9bc0-fd947c8c9942\") " Nov 03 00:13:03.310649 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.310621 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-openshift-ca" pod "e28074d5-66da-451a-9bc0-fd947c8c9942" (UID: "e28074d5-66da-451a-9bc0-fd947c8c9942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:13:03.312440 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.312404 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28074d5-66da-451a-9bc0-fd947c8c9942-kube-api-access-q8gmj" pod "e28074d5-66da-451a-9bc0-fd947c8c9942" (UID: "e28074d5-66da-451a-9bc0-fd947c8c9942"). InnerVolumeSpecName "kube-api-access-q8gmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:03.312588 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.312446 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-xks-ca" pod "e28074d5-66da-451a-9bc0-fd947c8c9942" (UID: "e28074d5-66da-451a-9bc0-fd947c8c9942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:13:03.412040 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.411958 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.412040 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.411988 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8gmj\" (UniqueName: \"kubernetes.io/projected/e28074d5-66da-451a-9bc0-fd947c8c9942-kube-api-access-q8gmj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.412040 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.412000 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28074d5-66da-451a-9bc0-fd947c8c9942-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.718032 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.718007 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:13:03.815462 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815423 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-token\" (UniqueName: \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-token\") pod \"0ff877c1-a472-40e6-9253-98ce9a9d2098\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " Nov 03 00:13:03.815644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815476 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-certs\") pod \"0ff877c1-a472-40e6-9253-98ce9a9d2098\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " Nov 03 00:13:03.815644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815541 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-data\") pod \"0ff877c1-a472-40e6-9253-98ce9a9d2098\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " Nov 03 00:13:03.815644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815576 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-credential-socket\") pod \"0ff877c1-a472-40e6-9253-98ce9a9d2098\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " Nov 03 00:13:03.815644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815614 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-envoy\") pod \"0ff877c1-a472-40e6-9253-98ce9a9d2098\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " Nov 03 00:13:03.815874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815690 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-podinfo\") pod \"0ff877c1-a472-40e6-9253-98ce9a9d2098\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " Nov 03 00:13:03.815874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815730 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff877c1-a472-40e6-9253-98ce9a9d2098-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff877c1-a472-40e6-9253-98ce9a9d2098-istiod-ca-cert\") pod \"0ff877c1-a472-40e6-9253-98ce9a9d2098\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " Nov 03 00:13:03.815874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815737 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-certs" pod "0ff877c1-a472-40e6-9253-98ce9a9d2098" (UID: "0ff877c1-a472-40e6-9253-98ce9a9d2098"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:03.815874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815768 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-socket\") pod \"0ff877c1-a472-40e6-9253-98ce9a9d2098\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " Nov 03 00:13:03.815874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815803 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-kube-api-access-nbx4z\" (UniqueName: \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-kube-api-access-nbx4z\") pod \"0ff877c1-a472-40e6-9253-98ce9a9d2098\" (UID: \"0ff877c1-a472-40e6-9253-98ce9a9d2098\") " Nov 03 00:13:03.816169 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815910 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-credential-socket" pod "0ff877c1-a472-40e6-9253-98ce9a9d2098" (UID: "0ff877c1-a472-40e6-9253-98ce9a9d2098"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:03.816169 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.815924 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-data" pod "0ff877c1-a472-40e6-9253-98ce9a9d2098" (UID: "0ff877c1-a472-40e6-9253-98ce9a9d2098"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:03.816314 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.816291 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-socket" pod "0ff877c1-a472-40e6-9253-98ce9a9d2098" (UID: "0ff877c1-a472-40e6-9253-98ce9a9d2098"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:03.816382 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.816228 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff877c1-a472-40e6-9253-98ce9a9d2098-istiod-ca-cert" pod "0ff877c1-a472-40e6-9253-98ce9a9d2098" (UID: "0ff877c1-a472-40e6-9253-98ce9a9d2098"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:13:03.816382 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.816318 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.816382 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.816338 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.816382 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.816352 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.818177 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.818134 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-kube-api-access-nbx4z" pod "0ff877c1-a472-40e6-9253-98ce9a9d2098" (UID: "0ff877c1-a472-40e6-9253-98ce9a9d2098"). InnerVolumeSpecName "kube-api-access-nbx4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:03.818297 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.818194 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-envoy" pod "0ff877c1-a472-40e6-9253-98ce9a9d2098" (UID: "0ff877c1-a472-40e6-9253-98ce9a9d2098"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:13:03.818380 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.818360 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-podinfo" pod "0ff877c1-a472-40e6-9253-98ce9a9d2098" (UID: "0ff877c1-a472-40e6-9253-98ce9a9d2098"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:13:03.818380 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.818372 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-token" pod "0ff877c1-a472-40e6-9253-98ce9a9d2098" (UID: "0ff877c1-a472-40e6-9253-98ce9a9d2098"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:03.917327 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.917295 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.917327 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.917321 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.917327 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.917330 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff877c1-a472-40e6-9253-98ce9a9d2098-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.917573 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.917341 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff877c1-a472-40e6-9253-98ce9a9d2098-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.917573 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.917350 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbx4z\" (UniqueName: \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-kube-api-access-nbx4z\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:03.917573 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:03.917361 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff877c1-a472-40e6-9253-98ce9a9d2098-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:04.114861 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.114826 5672 generic.go:356] "Generic (PLEG): container finished" podID="0ff877c1-a472-40e6-9253-98ce9a9d2098" containerID="20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c" exitCode=0 Nov 03 00:13:04.115048 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.114873 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" Nov 03 00:13:04.115104 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.115066 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" event={"ID":"0ff877c1-a472-40e6-9253-98ce9a9d2098","Type":"ContainerDied","Data":"20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c"} Nov 03 00:13:04.115164 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.115117 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb" event={"ID":"0ff877c1-a472-40e6-9253-98ce9a9d2098","Type":"ContainerDied","Data":"2727608d696759bfbc682802ef0393bb3d8ae454fb18323b1343b52ea2336253"} Nov 03 00:13:04.115208 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.115167 5672 scope.go:122] "RemoveContainer" containerID="20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c" Nov 03 00:13:04.118739 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.118719 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" Nov 03 00:13:04.118850 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.118719 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr" event={"ID":"e28074d5-66da-451a-9bc0-fd947c8c9942","Type":"ContainerDied","Data":"4299bb615fad8e2ee8e8a844e0b7996fb6462f9d0437fef60dd8a3f4346de24d"} Nov 03 00:13:04.118894 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.118848 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4299bb615fad8e2ee8e8a844e0b7996fb6462f9d0437fef60dd8a3f4346de24d" Nov 03 00:13:04.133914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.133889 5672 scope.go:122] "RemoveContainer" containerID="20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c" Nov 03 00:13:04.134274 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:13:04.134250 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c\": container with ID starting with 20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c not found: ID does not exist" containerID="20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c" Nov 03 00:13:04.134392 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.134281 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c"} err="failed to get container status \"20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c\": rpc error: code = NotFound desc = could not find container \"20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c\": container with ID starting with 20cedfa3abde74b96b4a102446fa480963d5720cade3da253b4607318c23c49c not found: ID does not exist" Nov 03 00:13:04.268481 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.268442 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb"] Nov 03 00:13:04.285500 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.285472 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6xgsb"] Nov 03 00:13:04.713006 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:04.712973 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff877c1-a472-40e6-9253-98ce9a9d2098" path="/var/lib/kubelet/pods/0ff877c1-a472-40e6-9253-98ce9a9d2098/volumes" Nov 03 00:13:10.046524 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.046445 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n"] Nov 03 00:13:10.046954 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.046941 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0ff877c1-a472-40e6-9253-98ce9a9d2098" containerName="istio-proxy" Nov 03 00:13:10.047011 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.046955 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff877c1-a472-40e6-9253-98ce9a9d2098" containerName="istio-proxy" Nov 03 00:13:10.047011 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.046967 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e28074d5-66da-451a-9bc0-fd947c8c9942" containerName="revoke-keys" Nov 03 00:13:10.047011 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.046972 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28074d5-66da-451a-9bc0-fd947c8c9942" containerName="revoke-keys" Nov 03 00:13:10.047112 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.047081 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff877c1-a472-40e6-9253-98ce9a9d2098" containerName="istio-proxy" Nov 03 00:13:10.047112 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.047091 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28074d5-66da-451a-9bc0-fd947c8c9942" containerName="revoke-keys" Nov 03 00:13:10.050291 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.050270 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:10.088176 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.088112 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c518153a-74d5-495f-a977-65e65885c42c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05ea45ea-mqb7n\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:10.088356 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.088253 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c518153a-74d5-495f-a977-65e65885c42c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05ea45ea-mqb7n\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:10.088356 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.088342 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gfxv\" (UniqueName: \"kubernetes.io/projected/c518153a-74d5-495f-a977-65e65885c42c-kube-api-access-2gfxv\") pod \"maas-api-revoke-sub-05ea45ea-mqb7n\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:10.102499 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.102458 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n"] Nov 03 00:13:10.145458 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.145414 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:13:10.145458 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.145452 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:13:10.145711 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.145455 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:13:10.145711 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.145458 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:13:10.189542 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.189499 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c518153a-74d5-495f-a977-65e65885c42c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05ea45ea-mqb7n\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:10.189745 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.189588 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gfxv\" (UniqueName: \"kubernetes.io/projected/c518153a-74d5-495f-a977-65e65885c42c-kube-api-access-2gfxv\") pod \"maas-api-revoke-sub-05ea45ea-mqb7n\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:10.189822 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.189759 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c518153a-74d5-495f-a977-65e65885c42c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05ea45ea-mqb7n\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:10.190414 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.190394 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c518153a-74d5-495f-a977-65e65885c42c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05ea45ea-mqb7n\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:10.191885 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.191869 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c518153a-74d5-495f-a977-65e65885c42c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05ea45ea-mqb7n\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:10.242428 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.242397 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gfxv\" (UniqueName: \"kubernetes.io/projected/c518153a-74d5-495f-a977-65e65885c42c-kube-api-access-2gfxv\") pod \"maas-api-revoke-sub-05ea45ea-mqb7n\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:10.363744 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.363661 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:10.737060 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:10.737035 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n"] Nov 03 00:13:10.738677 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:13:10.738643 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc518153a_74d5_495f_a977_65e65885c42c.slice/crio-36bb09b890ed3c1950d7a2786890b8e7e0b1494f9c7dc376767b98e5aebdf504 WatchSource:0}: Error finding container 36bb09b890ed3c1950d7a2786890b8e7e0b1494f9c7dc376767b98e5aebdf504: Status 404 returned error can't find the container with id 36bb09b890ed3c1950d7a2786890b8e7e0b1494f9c7dc376767b98e5aebdf504 Nov 03 00:13:11.167018 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:11.166981 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" event={"ID":"c518153a-74d5-495f-a977-65e65885c42c","Type":"ContainerStarted","Data":"8fce1acef2a8a5b0be56c2c881d9cdc2991016037c9cceefc5646030c6426b9f"} Nov 03 00:13:11.167485 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:11.167038 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" event={"ID":"c518153a-74d5-495f-a977-65e65885c42c","Type":"ContainerStarted","Data":"36bb09b890ed3c1950d7a2786890b8e7e0b1494f9c7dc376767b98e5aebdf504"} Nov 03 00:13:11.273971 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:11.273917 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" podStartSLOduration=1.273900195 podStartE2EDuration="1.273900195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:11.272367337 +0000 UTC m=+4784.679679166" watchObservedRunningTime="2026-11-03 00:13:11.273900195 +0000 UTC m=+4784.681212024" Nov 03 00:13:12.175200 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:12.175166 5672 generic.go:356] "Generic (PLEG): container finished" podID="c518153a-74d5-495f-a977-65e65885c42c" containerID="8fce1acef2a8a5b0be56c2c881d9cdc2991016037c9cceefc5646030c6426b9f" exitCode=0 Nov 03 00:13:12.175633 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:12.175205 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" event={"ID":"c518153a-74d5-495f-a977-65e65885c42c","Type":"ContainerDied","Data":"8fce1acef2a8a5b0be56c2c881d9cdc2991016037c9cceefc5646030c6426b9f"} Nov 03 00:13:13.326223 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:13.326200 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:13.430908 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:13.430876 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c518153a-74d5-495f-a977-65e65885c42c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c518153a-74d5-495f-a977-65e65885c42c-maas-api-xks-ca\") pod \"c518153a-74d5-495f-a977-65e65885c42c\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " Nov 03 00:13:13.431094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:13.431055 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c518153a-74d5-495f-a977-65e65885c42c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c518153a-74d5-495f-a977-65e65885c42c-maas-api-openshift-ca\") pod \"c518153a-74d5-495f-a977-65e65885c42c\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " Nov 03 00:13:13.431180 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:13.431093 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c518153a-74d5-495f-a977-65e65885c42c-kube-api-access-2gfxv\" (UniqueName: \"kubernetes.io/projected/c518153a-74d5-495f-a977-65e65885c42c-kube-api-access-2gfxv\") pod \"c518153a-74d5-495f-a977-65e65885c42c\" (UID: \"c518153a-74d5-495f-a977-65e65885c42c\") " Nov 03 00:13:13.432734 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:13.431576 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c518153a-74d5-495f-a977-65e65885c42c-maas-api-openshift-ca" pod "c518153a-74d5-495f-a977-65e65885c42c" (UID: "c518153a-74d5-495f-a977-65e65885c42c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:13:13.433612 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:13.433577 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c518153a-74d5-495f-a977-65e65885c42c-maas-api-xks-ca" pod "c518153a-74d5-495f-a977-65e65885c42c" (UID: "c518153a-74d5-495f-a977-65e65885c42c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:13:13.433733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:13.433663 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c518153a-74d5-495f-a977-65e65885c42c-kube-api-access-2gfxv" pod "c518153a-74d5-495f-a977-65e65885c42c" (UID: "c518153a-74d5-495f-a977-65e65885c42c"). InnerVolumeSpecName "kube-api-access-2gfxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:13:13.532684 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:13.532652 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c518153a-74d5-495f-a977-65e65885c42c-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:13.532684 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:13.532680 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c518153a-74d5-495f-a977-65e65885c42c-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:13.532684 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:13.532690 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gfxv\" (UniqueName: \"kubernetes.io/projected/c518153a-74d5-495f-a977-65e65885c42c-kube-api-access-2gfxv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:13:14.189233 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:14.189200 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" Nov 03 00:13:14.189233 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:14.189212 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n" event={"ID":"c518153a-74d5-495f-a977-65e65885c42c","Type":"ContainerDied","Data":"36bb09b890ed3c1950d7a2786890b8e7e0b1494f9c7dc376767b98e5aebdf504"} Nov 03 00:13:14.189450 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:14.189267 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36bb09b890ed3c1950d7a2786890b8e7e0b1494f9c7dc376767b98e5aebdf504" Nov 03 00:13:26.021921 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:13:26.021883 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:13:29.797543 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:29.797510 5672 scope.go:122] "RemoveContainer" containerID="0e4ce46d896b82146abe6c525a7e2c1b9fb2916dd1664d8f73921cb77c7b44d3" Nov 03 00:13:34.043256 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:34.043221 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn"] Nov 03 00:13:34.089641 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:34.089612 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a35c6d1-h88kn"] Nov 03 00:13:34.712908 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:34.712870 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="626f3019-c1dc-41cd-8b2b-03310f8b2e07" path="/var/lib/kubelet/pods/626f3019-c1dc-41cd-8b2b-03310f8b2e07/volumes" Nov 03 00:13:56.021054 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:13:56.021014 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:13:59.825592 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:13:59.825548 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:14:16.654376 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:16.654324 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr"] Nov 03 00:14:16.809661 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:16.809627 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8"] Nov 03 00:14:21.655285 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:21.655226 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" podUID="1ace1b3c-df89-4751-96a3-0899c15c7d86" containerName="payload-processing" containerID="cri-o://0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64" gracePeriod=25 Nov 03 00:14:21.810292 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:21.810242 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" podUID="551678ad-8f0d-4350-a6b8-b6c48714ec9c" containerName="payload-pre-processing" containerID="cri-o://f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4" gracePeriod=25 Nov 03 00:14:21.939933 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:21.939905 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:14:22.067305 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.067279 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:14:22.126837 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.126804 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ace1b3c-df89-4751-96a3-0899c15c7d86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ace1b3c-df89-4751-96a3-0899c15c7d86-plugins-config-volume\") pod \"1ace1b3c-df89-4751-96a3-0899c15c7d86\" (UID: \"1ace1b3c-df89-4751-96a3-0899c15c7d86\") " Nov 03 00:14:22.127010 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.126916 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ace1b3c-df89-4751-96a3-0899c15c7d86-kube-api-access-df42w\" (UniqueName: \"kubernetes.io/projected/1ace1b3c-df89-4751-96a3-0899c15c7d86-kube-api-access-df42w\") pod \"1ace1b3c-df89-4751-96a3-0899c15c7d86\" (UID: \"1ace1b3c-df89-4751-96a3-0899c15c7d86\") " Nov 03 00:14:22.127300 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.127275 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ace1b3c-df89-4751-96a3-0899c15c7d86-plugins-config-volume" pod "1ace1b3c-df89-4751-96a3-0899c15c7d86" (UID: "1ace1b3c-df89-4751-96a3-0899c15c7d86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:22.129001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.128978 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ace1b3c-df89-4751-96a3-0899c15c7d86-kube-api-access-df42w" pod "1ace1b3c-df89-4751-96a3-0899c15c7d86" (UID: "1ace1b3c-df89-4751-96a3-0899c15c7d86"). InnerVolumeSpecName "kube-api-access-df42w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:22.228458 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.228421 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551678ad-8f0d-4350-a6b8-b6c48714ec9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/551678ad-8f0d-4350-a6b8-b6c48714ec9c-plugins-config-volume\") pod \"551678ad-8f0d-4350-a6b8-b6c48714ec9c\" (UID: \"551678ad-8f0d-4350-a6b8-b6c48714ec9c\") " Nov 03 00:14:22.228670 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.228492 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551678ad-8f0d-4350-a6b8-b6c48714ec9c-kube-api-access-xttrd\" (UniqueName: \"kubernetes.io/projected/551678ad-8f0d-4350-a6b8-b6c48714ec9c-kube-api-access-xttrd\") pod \"551678ad-8f0d-4350-a6b8-b6c48714ec9c\" (UID: \"551678ad-8f0d-4350-a6b8-b6c48714ec9c\") " Nov 03 00:14:22.228885 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.228855 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551678ad-8f0d-4350-a6b8-b6c48714ec9c-plugins-config-volume" pod "551678ad-8f0d-4350-a6b8-b6c48714ec9c" (UID: "551678ad-8f0d-4350-a6b8-b6c48714ec9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:22.229067 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.229052 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/551678ad-8f0d-4350-a6b8-b6c48714ec9c-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:22.229132 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.229070 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ace1b3c-df89-4751-96a3-0899c15c7d86-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:22.229132 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.229081 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df42w\" (UniqueName: \"kubernetes.io/projected/1ace1b3c-df89-4751-96a3-0899c15c7d86-kube-api-access-df42w\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:22.230572 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.230543 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551678ad-8f0d-4350-a6b8-b6c48714ec9c-kube-api-access-xttrd" pod "551678ad-8f0d-4350-a6b8-b6c48714ec9c" (UID: "551678ad-8f0d-4350-a6b8-b6c48714ec9c"). InnerVolumeSpecName "kube-api-access-xttrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:22.329792 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.329757 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xttrd\" (UniqueName: \"kubernetes.io/projected/551678ad-8f0d-4350-a6b8-b6c48714ec9c-kube-api-access-xttrd\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:22.656714 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.656615 5672 generic.go:356] "Generic (PLEG): container finished" podID="551678ad-8f0d-4350-a6b8-b6c48714ec9c" containerID="f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4" exitCode=0 Nov 03 00:14:22.656714 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.656674 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" Nov 03 00:14:22.657269 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.656713 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" event={"ID":"551678ad-8f0d-4350-a6b8-b6c48714ec9c","Type":"ContainerDied","Data":"f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4"} Nov 03 00:14:22.657269 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.656766 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8" event={"ID":"551678ad-8f0d-4350-a6b8-b6c48714ec9c","Type":"ContainerDied","Data":"ce42d5920b90d0aa2a7829aec1ea40e015b3fd0f855b36de1e5a305ce21e63a9"} Nov 03 00:14:22.657269 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.656796 5672 scope.go:122] "RemoveContainer" containerID="f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4" Nov 03 00:14:22.658408 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.658383 5672 generic.go:356] "Generic (PLEG): container finished" podID="1ace1b3c-df89-4751-96a3-0899c15c7d86" containerID="0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64" exitCode=0 Nov 03 00:14:22.658495 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.658420 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" event={"ID":"1ace1b3c-df89-4751-96a3-0899c15c7d86","Type":"ContainerDied","Data":"0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64"} Nov 03 00:14:22.658495 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.658434 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" Nov 03 00:14:22.658495 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.658459 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr" event={"ID":"1ace1b3c-df89-4751-96a3-0899c15c7d86","Type":"ContainerDied","Data":"521a671ad2094791c0e6588c1d6b533bb0566acad082ceb41f008debbad02dd9"} Nov 03 00:14:22.668204 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.668169 5672 scope.go:122] "RemoveContainer" containerID="f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4" Nov 03 00:14:22.668477 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:14:22.668457 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4\": container with ID starting with f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4 not found: ID does not exist" containerID="f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4" Nov 03 00:14:22.668576 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.668484 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4"} err="failed to get container status \"f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4\": rpc error: code = NotFound desc = could not find container \"f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4\": container with ID starting with f27a43b3d7dad4d3a7cb7d48837d95dd435bd486e1a28db4e36fad27518e36d4 not found: ID does not exist" Nov 03 00:14:22.668576 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.668500 5672 scope.go:122] "RemoveContainer" containerID="0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64" Nov 03 00:14:22.682853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.682836 5672 scope.go:122] "RemoveContainer" containerID="0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64" Nov 03 00:14:22.683116 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:14:22.683089 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64\": container with ID starting with 0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64 not found: ID does not exist" containerID="0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64" Nov 03 00:14:22.683222 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.683126 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64"} err="failed to get container status \"0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64\": rpc error: code = NotFound desc = could not find container \"0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64\": container with ID starting with 0a1be3b76b403c372ccf49096f0f2e862ebad135a52cb7f7e1ab45ea4ff5db64 not found: ID does not exist" Nov 03 00:14:22.810794 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.810758 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr"] Nov 03 00:14:22.858698 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.858663 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f668443-5844d9f575-mfmdr"] Nov 03 00:14:22.958022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.957935 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8"] Nov 03 00:14:22.963834 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:22.963804 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1f668443-7cd89d6479-xrnx8"] Nov 03 00:14:24.712753 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:24.712720 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ace1b3c-df89-4751-96a3-0899c15c7d86" path="/var/lib/kubelet/pods/1ace1b3c-df89-4751-96a3-0899c15c7d86/volumes" Nov 03 00:14:24.713134 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:24.713120 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551678ad-8f0d-4350-a6b8-b6c48714ec9c" path="/var/lib/kubelet/pods/551678ad-8f0d-4350-a6b8-b6c48714ec9c/volumes" Nov 03 00:14:26.022082 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:14:26.021982 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:14:29.829001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:29.828957 5672 scope.go:122] "RemoveContainer" containerID="cddfc06e59e11ad9e9a74b153820bd12e417d3e8b64990b8a108329d5d5a8f1e" Nov 03 00:14:30.955776 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:30.955736 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42"] Nov 03 00:14:30.956242 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:30.955964 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" containerID="cri-o://603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43" gracePeriod=30 Nov 03 00:14:34.116885 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:34.116848 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qcj42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:14:34.117311 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:34.116921 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:14:36.226456 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.226385 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:14:36.281407 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281378 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istiod-ca-cert\") pod \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " Nov 03 00:14:36.281574 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281431 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-podinfo\") pod \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " Nov 03 00:14:36.281574 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281456 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-envoy\") pod \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " Nov 03 00:14:36.281574 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281491 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-token\" (UniqueName: \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-token\") pod \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " Nov 03 00:14:36.281574 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281525 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-credential-socket\") pod \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " Nov 03 00:14:36.281574 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281560 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-socket\") pod \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " Nov 03 00:14:36.281881 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281649 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-certs\") pod \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " Nov 03 00:14:36.281881 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281736 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-kube-api-access-d79qj\" (UniqueName: \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-kube-api-access-d79qj\") pod \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " Nov 03 00:14:36.281881 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281772 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-data\") pod \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\" (UID: \"a39ffff0-85c2-44c0-b5c6-2b8dbb779079\") " Nov 03 00:14:36.282058 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281811 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istiod-ca-cert" pod "a39ffff0-85c2-44c0-b5c6-2b8dbb779079" (UID: "a39ffff0-85c2-44c0-b5c6-2b8dbb779079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:36.282058 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281817 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-credential-socket" pod "a39ffff0-85c2-44c0-b5c6-2b8dbb779079" (UID: "a39ffff0-85c2-44c0-b5c6-2b8dbb779079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.282058 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281901 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-socket" pod "a39ffff0-85c2-44c0-b5c6-2b8dbb779079" (UID: "a39ffff0-85c2-44c0-b5c6-2b8dbb779079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.282058 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.281945 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-certs" pod "a39ffff0-85c2-44c0-b5c6-2b8dbb779079" (UID: "a39ffff0-85c2-44c0-b5c6-2b8dbb779079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.282283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.282116 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-data" pod "a39ffff0-85c2-44c0-b5c6-2b8dbb779079" (UID: "a39ffff0-85c2-44c0-b5c6-2b8dbb779079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.282359 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.282320 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.282359 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.282356 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.282474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.282371 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.282474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.282387 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.282474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.282401 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.283858 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.283830 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-kube-api-access-d79qj" pod "a39ffff0-85c2-44c0-b5c6-2b8dbb779079" (UID: "a39ffff0-85c2-44c0-b5c6-2b8dbb779079"). InnerVolumeSpecName "kube-api-access-d79qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:36.283991 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.283877 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-token" pod "a39ffff0-85c2-44c0-b5c6-2b8dbb779079" (UID: "a39ffff0-85c2-44c0-b5c6-2b8dbb779079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:36.284230 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.284209 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-envoy" pod "a39ffff0-85c2-44c0-b5c6-2b8dbb779079" (UID: "a39ffff0-85c2-44c0-b5c6-2b8dbb779079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:14:36.284342 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.284320 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-podinfo" pod "a39ffff0-85c2-44c0-b5c6-2b8dbb779079" (UID: "a39ffff0-85c2-44c0-b5c6-2b8dbb779079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:14:36.383462 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.383418 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.383462 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.383467 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.383725 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.383483 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d79qj\" (UniqueName: \"kubernetes.io/projected/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-kube-api-access-d79qj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.383725 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.383498 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39ffff0-85c2-44c0-b5c6-2b8dbb779079-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:36.753811 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.753770 5672 generic.go:356] "Generic (PLEG): container finished" podID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerID="603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43" exitCode=0 Nov 03 00:14:36.754003 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.753828 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" Nov 03 00:14:36.754003 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.753850 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" event={"ID":"a39ffff0-85c2-44c0-b5c6-2b8dbb779079","Type":"ContainerDied","Data":"603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43"} Nov 03 00:14:36.754003 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.753913 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42" event={"ID":"a39ffff0-85c2-44c0-b5c6-2b8dbb779079","Type":"ContainerDied","Data":"92b9ea8d65f83ca45c22cc72b4ac4f1852263f38d346f8ad88432a88463fe12f"} Nov 03 00:14:36.754003 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.753946 5672 scope.go:122] "RemoveContainer" containerID="603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43" Nov 03 00:14:36.766408 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.766386 5672 scope.go:122] "RemoveContainer" containerID="603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43" Nov 03 00:14:36.766733 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:14:36.766705 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43\": container with ID starting with 603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43 not found: ID does not exist" containerID="603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43" Nov 03 00:14:36.766825 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.766743 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43"} err="failed to get container status \"603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43\": rpc error: code = NotFound desc = could not find container \"603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43\": container with ID starting with 603acffd0149080b0bea2dd02487ea4d61446d8c297af9dc86e9aa0f2d344f43 not found: ID does not exist" Nov 03 00:14:36.908486 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.908454 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42"] Nov 03 00:14:36.955103 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:36.955072 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qcj42"] Nov 03 00:14:37.924623 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:37.924590 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:14:38.713406 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:38.713372 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" path="/var/lib/kubelet/pods/a39ffff0-85c2-44c0-b5c6-2b8dbb779079/volumes" Nov 03 00:14:48.987224 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987183 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c"] Nov 03 00:14:48.987732 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987716 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c518153a-74d5-495f-a977-65e65885c42c" containerName="revoke-keys" Nov 03 00:14:48.987798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987733 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c518153a-74d5-495f-a977-65e65885c42c" containerName="revoke-keys" Nov 03 00:14:48.987798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987748 5672 cpu_manager.go:414] "containerMap: removing container" podUID="1ace1b3c-df89-4751-96a3-0899c15c7d86" containerName="payload-processing" Nov 03 00:14:48.987798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987753 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ace1b3c-df89-4751-96a3-0899c15c7d86" containerName="payload-processing" Nov 03 00:14:48.987798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987780 5672 cpu_manager.go:414] "containerMap: removing container" podUID="551678ad-8f0d-4350-a6b8-b6c48714ec9c" containerName="payload-pre-processing" Nov 03 00:14:48.987798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987785 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551678ad-8f0d-4350-a6b8-b6c48714ec9c" containerName="payload-pre-processing" Nov 03 00:14:48.987798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987797 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" Nov 03 00:14:48.988123 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987802 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" Nov 03 00:14:48.988123 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987872 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c518153a-74d5-495f-a977-65e65885c42c" containerName="revoke-keys" Nov 03 00:14:48.988123 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987881 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ace1b3c-df89-4751-96a3-0899c15c7d86" containerName="payload-processing" Nov 03 00:14:48.988123 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987891 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="551678ad-8f0d-4350-a6b8-b6c48714ec9c" containerName="payload-pre-processing" Nov 03 00:14:48.988123 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.987897 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39ffff0-85c2-44c0-b5c6-2b8dbb779079" containerName="istio-proxy" Nov 03 00:14:48.991366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:48.991345 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:49.041506 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.041477 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:14:49.041733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.041591 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:14:49.087519 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.087489 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:14:49.087784 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.087765 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:14:49.090513 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.090493 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c"] Nov 03 00:14:49.129667 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.129642 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422ce4fb-m6t4c\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:49.129783 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.129685 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422ce4fb-m6t4c\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:49.129893 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.129874 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvs4q\" (UniqueName: \"kubernetes.io/projected/1df505a7-36de-4169-a32b-9fba92b528aa-kube-api-access-kvs4q\") pod \"maas-api-revoke-sub-422ce4fb-m6t4c\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:49.231097 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.231059 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422ce4fb-m6t4c\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:49.231320 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.231111 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422ce4fb-m6t4c\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:49.231320 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.231225 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvs4q\" (UniqueName: \"kubernetes.io/projected/1df505a7-36de-4169-a32b-9fba92b528aa-kube-api-access-kvs4q\") pod \"maas-api-revoke-sub-422ce4fb-m6t4c\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:49.231794 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.231766 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422ce4fb-m6t4c\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:49.233715 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.233688 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422ce4fb-m6t4c\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:49.285856 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.285780 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvs4q\" (UniqueName: \"kubernetes.io/projected/1df505a7-36de-4169-a32b-9fba92b528aa-kube-api-access-kvs4q\") pod \"maas-api-revoke-sub-422ce4fb-m6t4c\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:49.305136 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.305108 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:49.681125 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.681095 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c"] Nov 03 00:14:49.683682 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:14:49.683644 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df505a7_36de_4169_a32b_9fba92b528aa.slice/crio-60fef887d8495ef9d4aee8ef41c1c67b9be5694041924acfcbe4c95d4c1d6db1 WatchSource:0}: Error finding container 60fef887d8495ef9d4aee8ef41c1c67b9be5694041924acfcbe4c95d4c1d6db1: Status 404 returned error can't find the container with id 60fef887d8495ef9d4aee8ef41c1c67b9be5694041924acfcbe4c95d4c1d6db1 Nov 03 00:14:49.841342 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:49.841306 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" event={"ID":"1df505a7-36de-4169-a32b-9fba92b528aa","Type":"ContainerStarted","Data":"60fef887d8495ef9d4aee8ef41c1c67b9be5694041924acfcbe4c95d4c1d6db1"} Nov 03 00:14:50.858639 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:50.858536 5672 generic.go:356] "Generic (PLEG): container finished" podID="1df505a7-36de-4169-a32b-9fba92b528aa" containerID="041e26711ebfc181ad32b0dcbd7388737d62638b01904e79f580f3d015c1c34c" exitCode=0 Nov 03 00:14:50.859008 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:50.858624 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" event={"ID":"1df505a7-36de-4169-a32b-9fba92b528aa","Type":"ContainerDied","Data":"041e26711ebfc181ad32b0dcbd7388737d62638b01904e79f580f3d015c1c34c"} Nov 03 00:14:52.014970 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.014939 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:52.168938 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.168858 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df505a7-36de-4169-a32b-9fba92b528aa-kube-api-access-kvs4q\" (UniqueName: \"kubernetes.io/projected/1df505a7-36de-4169-a32b-9fba92b528aa-kube-api-access-kvs4q\") pod \"1df505a7-36de-4169-a32b-9fba92b528aa\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " Nov 03 00:14:52.168938 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.168908 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-openshift-ca\") pod \"1df505a7-36de-4169-a32b-9fba92b528aa\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " Nov 03 00:14:52.169265 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.169080 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-xks-ca\") pod \"1df505a7-36de-4169-a32b-9fba92b528aa\" (UID: \"1df505a7-36de-4169-a32b-9fba92b528aa\") " Nov 03 00:14:52.169340 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.169302 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-openshift-ca" pod "1df505a7-36de-4169-a32b-9fba92b528aa" (UID: "1df505a7-36de-4169-a32b-9fba92b528aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:14:52.171048 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.171020 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df505a7-36de-4169-a32b-9fba92b528aa-kube-api-access-kvs4q" pod "1df505a7-36de-4169-a32b-9fba92b528aa" (UID: "1df505a7-36de-4169-a32b-9fba92b528aa"). InnerVolumeSpecName "kube-api-access-kvs4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:14:52.171189 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.171100 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-xks-ca" pod "1df505a7-36de-4169-a32b-9fba92b528aa" (UID: "1df505a7-36de-4169-a32b-9fba92b528aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:14:52.270012 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.269976 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:52.270012 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.270004 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvs4q\" (UniqueName: \"kubernetes.io/projected/1df505a7-36de-4169-a32b-9fba92b528aa-kube-api-access-kvs4q\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:52.270012 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.270014 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df505a7-36de-4169-a32b-9fba92b528aa-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:14:52.880848 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.880821 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" Nov 03 00:14:52.880848 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.880833 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c" event={"ID":"1df505a7-36de-4169-a32b-9fba92b528aa","Type":"ContainerDied","Data":"60fef887d8495ef9d4aee8ef41c1c67b9be5694041924acfcbe4c95d4c1d6db1"} Nov 03 00:14:52.881127 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:52.880885 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60fef887d8495ef9d4aee8ef41c1c67b9be5694041924acfcbe4c95d4c1d6db1" Nov 03 00:14:56.020707 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:14:56.020667 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:14:56.729235 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:56.729202 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl"] Nov 03 00:14:59.660090 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.660057 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6"] Nov 03 00:14:59.660620 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.660604 5672 cpu_manager.go:414] "containerMap: removing container" podUID="1df505a7-36de-4169-a32b-9fba92b528aa" containerName="revoke-keys" Nov 03 00:14:59.660666 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.660622 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df505a7-36de-4169-a32b-9fba92b528aa" containerName="revoke-keys" Nov 03 00:14:59.660761 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.660750 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df505a7-36de-4169-a32b-9fba92b528aa" containerName="revoke-keys" Nov 03 00:14:59.664324 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.664305 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:14:59.714389 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.714361 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:14:59.749790 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.749762 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242382cb-cqss6\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:14:59.749959 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.749915 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242382cb-cqss6\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:14:59.750021 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.749981 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cwtt\" (UniqueName: \"kubernetes.io/projected/2144ab98-3fe5-4849-b80c-507d4e015a46-kube-api-access-6cwtt\") pod \"maas-api-revoke-sub-242382cb-cqss6\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:14:59.760566 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.760541 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:14:59.760715 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.760690 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:14:59.760786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.760742 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:14:59.763328 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.763307 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6"] Nov 03 00:14:59.850854 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.850821 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242382cb-cqss6\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:14:59.851066 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.850870 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cwtt\" (UniqueName: \"kubernetes.io/projected/2144ab98-3fe5-4849-b80c-507d4e015a46-kube-api-access-6cwtt\") pod \"maas-api-revoke-sub-242382cb-cqss6\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:14:59.851066 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.850961 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242382cb-cqss6\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:14:59.851807 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.851776 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242382cb-cqss6\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:14:59.853513 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.853483 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242382cb-cqss6\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:14:59.903659 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.903632 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cwtt\" (UniqueName: \"kubernetes.io/projected/2144ab98-3fe5-4849-b80c-507d4e015a46-kube-api-access-6cwtt\") pod \"maas-api-revoke-sub-242382cb-cqss6\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:14:59.978512 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:14:59.978485 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:15:00.153663 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.153631 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589"] Nov 03 00:15:00.158942 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.158919 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" Nov 03 00:15:00.160499 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.160465 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8"] Nov 03 00:15:00.166072 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.164989 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts"] Nov 03 00:15:00.166072 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.165464 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" Nov 03 00:15:00.169655 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.169632 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g"] Nov 03 00:15:00.169783 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.169709 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" Nov 03 00:15:00.173931 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.173913 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m"] Nov 03 00:15:00.174029 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.174001 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" Nov 03 00:15:00.178078 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.178056 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" Nov 03 00:15:00.181813 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.181791 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp"] Nov 03 00:15:00.185868 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.185848 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" Nov 03 00:15:00.190420 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.190399 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc"] Nov 03 00:15:00.195196 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.195176 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" Nov 03 00:15:00.203972 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.203621 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg"] Nov 03 00:15:00.211006 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.209340 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589"] Nov 03 00:15:00.211006 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.209439 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" Nov 03 00:15:00.256062 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.253713 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts"] Nov 03 00:15:00.258089 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258059 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwlgv\" (UniqueName: \"kubernetes.io/projected/583aad32-dafe-46d4-861f-f07566330db8-kube-api-access-fwlgv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8\" (UID: \"583aad32-dafe-46d4-861f-f07566330db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" Nov 03 00:15:00.258433 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258406 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589\" (UID: \"d82d527a-412c-4e3d-a8fe-afc3557f9ee5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" Nov 03 00:15:00.258559 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258454 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtr5\" (UniqueName: \"kubernetes.io/projected/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-kube-api-access-7qtr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589\" (UID: \"d82d527a-412c-4e3d-a8fe-afc3557f9ee5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" Nov 03 00:15:00.258559 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258480 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75lmb\" (UniqueName: \"kubernetes.io/projected/74c90ca2-20e4-4c85-94de-9e7127186ea9-kube-api-access-75lmb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts\" (UID: \"74c90ca2-20e4-4c85-94de-9e7127186ea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" Nov 03 00:15:00.258559 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258512 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186952f-5082-4cf7-bad8-a6ae7c4089b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc\" (UID: \"f186952f-5082-4cf7-bad8-a6ae7c4089b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" Nov 03 00:15:00.258559 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258547 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/583aad32-dafe-46d4-861f-f07566330db8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8\" (UID: \"583aad32-dafe-46d4-861f-f07566330db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" Nov 03 00:15:00.258812 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258618 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59278e2-911a-426e-80dc-7ed972e6fd36-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-hpvgg\" (UID: \"d59278e2-911a-426e-80dc-7ed972e6fd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" Nov 03 00:15:00.258812 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258657 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c80c505-2325-46ed-bdb0-95899c8ec5d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g\" (UID: \"3c80c505-2325-46ed-bdb0-95899c8ec5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" Nov 03 00:15:00.258812 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258681 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wkft\" (UniqueName: \"kubernetes.io/projected/f186952f-5082-4cf7-bad8-a6ae7c4089b5-kube-api-access-6wkft\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc\" (UID: \"f186952f-5082-4cf7-bad8-a6ae7c4089b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" Nov 03 00:15:00.258812 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258713 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m\" (UID: \"7056ae18-f4aa-4cb5-b53b-70bd53e7c477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" Nov 03 00:15:00.258812 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258769 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bzwg\" (UniqueName: \"kubernetes.io/projected/d59278e2-911a-426e-80dc-7ed972e6fd36-kube-api-access-2bzwg\") pod \"maas-api-key-cleanup-29894415-hpvgg\" (UID: \"d59278e2-911a-426e-80dc-7ed972e6fd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" Nov 03 00:15:00.258812 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258807 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfz4d\" (UniqueName: \"kubernetes.io/projected/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-kube-api-access-qfz4d\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m\" (UID: \"7056ae18-f4aa-4cb5-b53b-70bd53e7c477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" Nov 03 00:15:00.259114 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.258831 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp\" (UID: \"acb684d2-d193-4bf7-a07d-4e9c1d4116b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" Nov 03 00:15:00.259385 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.259366 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m"] Nov 03 00:15:00.263517 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.263495 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g"] Nov 03 00:15:00.266746 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.266723 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8"] Nov 03 00:15:00.272318 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.272294 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89tc\" (UniqueName: \"kubernetes.io/projected/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-kube-api-access-p89tc\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp\" (UID: \"acb684d2-d193-4bf7-a07d-4e9c1d4116b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" Nov 03 00:15:00.272505 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.272486 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42cnl\" (UniqueName: \"kubernetes.io/projected/3c80c505-2325-46ed-bdb0-95899c8ec5d8-kube-api-access-42cnl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g\" (UID: \"3c80c505-2325-46ed-bdb0-95899c8ec5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" Nov 03 00:15:00.272631 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.272617 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c90ca2-20e4-4c85-94de-9e7127186ea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts\" (UID: \"74c90ca2-20e4-4c85-94de-9e7127186ea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" Nov 03 00:15:00.273052 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.273031 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp"] Nov 03 00:15:00.321332 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.321300 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg"] Nov 03 00:15:00.325848 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.325807 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc"] Nov 03 00:15:00.364573 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:15:00.364542 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2144ab98_3fe5_4849_b80c_507d4e015a46.slice/crio-05cc4ede470d0be84d724bacc5bbb654fa72326d1a62c3deb5e569c309bda1d1 WatchSource:0}: Error finding container 05cc4ede470d0be84d724bacc5bbb654fa72326d1a62c3deb5e569c309bda1d1: Status 404 returned error can't find the container with id 05cc4ede470d0be84d724bacc5bbb654fa72326d1a62c3deb5e569c309bda1d1 Nov 03 00:15:00.365721 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.365637 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6"] Nov 03 00:15:00.375903 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.375870 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c80c505-2325-46ed-bdb0-95899c8ec5d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g\" (UID: \"3c80c505-2325-46ed-bdb0-95899c8ec5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" Nov 03 00:15:00.376000 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.375919 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wkft\" (UniqueName: \"kubernetes.io/projected/f186952f-5082-4cf7-bad8-a6ae7c4089b5-kube-api-access-6wkft\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc\" (UID: \"f186952f-5082-4cf7-bad8-a6ae7c4089b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" Nov 03 00:15:00.376000 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.375952 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m\" (UID: \"7056ae18-f4aa-4cb5-b53b-70bd53e7c477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" Nov 03 00:15:00.376000 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.375977 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bzwg\" (UniqueName: \"kubernetes.io/projected/d59278e2-911a-426e-80dc-7ed972e6fd36-kube-api-access-2bzwg\") pod \"maas-api-key-cleanup-29894415-hpvgg\" (UID: \"d59278e2-911a-426e-80dc-7ed972e6fd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" Nov 03 00:15:00.376130 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376013 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfz4d\" (UniqueName: \"kubernetes.io/projected/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-kube-api-access-qfz4d\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m\" (UID: \"7056ae18-f4aa-4cb5-b53b-70bd53e7c477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" Nov 03 00:15:00.376130 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376038 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp\" (UID: \"acb684d2-d193-4bf7-a07d-4e9c1d4116b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" Nov 03 00:15:00.376130 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376092 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89tc\" (UniqueName: \"kubernetes.io/projected/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-kube-api-access-p89tc\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp\" (UID: \"acb684d2-d193-4bf7-a07d-4e9c1d4116b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" Nov 03 00:15:00.376260 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376133 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42cnl\" (UniqueName: \"kubernetes.io/projected/3c80c505-2325-46ed-bdb0-95899c8ec5d8-kube-api-access-42cnl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g\" (UID: \"3c80c505-2325-46ed-bdb0-95899c8ec5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" Nov 03 00:15:00.376260 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376197 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c90ca2-20e4-4c85-94de-9e7127186ea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts\" (UID: \"74c90ca2-20e4-4c85-94de-9e7127186ea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" Nov 03 00:15:00.376335 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376284 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwlgv\" (UniqueName: \"kubernetes.io/projected/583aad32-dafe-46d4-861f-f07566330db8-kube-api-access-fwlgv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8\" (UID: \"583aad32-dafe-46d4-861f-f07566330db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" Nov 03 00:15:00.376335 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376318 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589\" (UID: \"d82d527a-412c-4e3d-a8fe-afc3557f9ee5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" Nov 03 00:15:00.376413 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376356 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtr5\" (UniqueName: \"kubernetes.io/projected/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-kube-api-access-7qtr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589\" (UID: \"d82d527a-412c-4e3d-a8fe-afc3557f9ee5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" Nov 03 00:15:00.376413 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376387 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75lmb\" (UniqueName: \"kubernetes.io/projected/74c90ca2-20e4-4c85-94de-9e7127186ea9-kube-api-access-75lmb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts\" (UID: \"74c90ca2-20e4-4c85-94de-9e7127186ea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" Nov 03 00:15:00.376490 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376427 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186952f-5082-4cf7-bad8-a6ae7c4089b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc\" (UID: \"f186952f-5082-4cf7-bad8-a6ae7c4089b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" Nov 03 00:15:00.376490 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376469 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/583aad32-dafe-46d4-861f-f07566330db8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8\" (UID: \"583aad32-dafe-46d4-861f-f07566330db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" Nov 03 00:15:00.376567 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.376553 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59278e2-911a-426e-80dc-7ed972e6fd36-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-hpvgg\" (UID: \"d59278e2-911a-426e-80dc-7ed972e6fd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" Nov 03 00:15:00.377298 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.377278 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59278e2-911a-426e-80dc-7ed972e6fd36-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-hpvgg\" (UID: \"d59278e2-911a-426e-80dc-7ed972e6fd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" Nov 03 00:15:00.377736 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.377719 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c80c505-2325-46ed-bdb0-95899c8ec5d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g\" (UID: \"3c80c505-2325-46ed-bdb0-95899c8ec5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" Nov 03 00:15:00.378391 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.378374 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m\" (UID: \"7056ae18-f4aa-4cb5-b53b-70bd53e7c477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" Nov 03 00:15:00.379097 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.379070 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp\" (UID: \"acb684d2-d193-4bf7-a07d-4e9c1d4116b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" Nov 03 00:15:00.379983 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.379961 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c90ca2-20e4-4c85-94de-9e7127186ea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts\" (UID: \"74c90ca2-20e4-4c85-94de-9e7127186ea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" Nov 03 00:15:00.380312 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.380288 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186952f-5082-4cf7-bad8-a6ae7c4089b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc\" (UID: \"f186952f-5082-4cf7-bad8-a6ae7c4089b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" Nov 03 00:15:00.380665 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.380642 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589\" (UID: \"d82d527a-412c-4e3d-a8fe-afc3557f9ee5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" Nov 03 00:15:00.380939 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.380919 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/583aad32-dafe-46d4-861f-f07566330db8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8\" (UID: \"583aad32-dafe-46d4-861f-f07566330db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" Nov 03 00:15:00.432497 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.432466 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wkft\" (UniqueName: \"kubernetes.io/projected/f186952f-5082-4cf7-bad8-a6ae7c4089b5-kube-api-access-6wkft\") pod \"maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc\" (UID: \"f186952f-5082-4cf7-bad8-a6ae7c4089b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" Nov 03 00:15:00.432630 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.432577 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfz4d\" (UniqueName: \"kubernetes.io/projected/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-kube-api-access-qfz4d\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m\" (UID: \"7056ae18-f4aa-4cb5-b53b-70bd53e7c477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" Nov 03 00:15:00.432768 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.432752 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bzwg\" (UniqueName: \"kubernetes.io/projected/d59278e2-911a-426e-80dc-7ed972e6fd36-kube-api-access-2bzwg\") pod \"maas-api-key-cleanup-29894415-hpvgg\" (UID: \"d59278e2-911a-426e-80dc-7ed972e6fd36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" Nov 03 00:15:00.468511 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.468454 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwlgv\" (UniqueName: \"kubernetes.io/projected/583aad32-dafe-46d4-861f-f07566330db8-kube-api-access-fwlgv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8\" (UID: \"583aad32-dafe-46d4-861f-f07566330db8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" Nov 03 00:15:00.468655 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.468575 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75lmb\" (UniqueName: \"kubernetes.io/projected/74c90ca2-20e4-4c85-94de-9e7127186ea9-kube-api-access-75lmb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts\" (UID: \"74c90ca2-20e4-4c85-94de-9e7127186ea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" Nov 03 00:15:00.468655 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.468635 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtr5\" (UniqueName: \"kubernetes.io/projected/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-kube-api-access-7qtr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589\" (UID: \"d82d527a-412c-4e3d-a8fe-afc3557f9ee5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" Nov 03 00:15:00.469082 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.469056 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89tc\" (UniqueName: \"kubernetes.io/projected/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-kube-api-access-p89tc\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp\" (UID: \"acb684d2-d193-4bf7-a07d-4e9c1d4116b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" Nov 03 00:15:00.469583 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.469562 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42cnl\" (UniqueName: \"kubernetes.io/projected/3c80c505-2325-46ed-bdb0-95899c8ec5d8-kube-api-access-42cnl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g\" (UID: \"3c80c505-2325-46ed-bdb0-95899c8ec5d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" Nov 03 00:15:00.476411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.476380 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" Nov 03 00:15:00.484209 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.484189 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" Nov 03 00:15:00.492802 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.492780 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" Nov 03 00:15:00.500437 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.500367 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" Nov 03 00:15:00.511460 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.511422 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" Nov 03 00:15:00.525178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.524100 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" Nov 03 00:15:00.534810 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.534635 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" Nov 03 00:15:00.544912 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.544742 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" Nov 03 00:15:00.940465 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.939825 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" event={"ID":"2144ab98-3fe5-4849-b80c-507d4e015a46","Type":"ContainerStarted","Data":"867ca72a64b95bf3a2e18161a648f1f76a93fef47986594d79618faace5be2cf"} Nov 03 00:15:00.940465 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:00.939889 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" event={"ID":"2144ab98-3fe5-4849-b80c-507d4e015a46","Type":"ContainerStarted","Data":"05cc4ede470d0be84d724bacc5bbb654fa72326d1a62c3deb5e569c309bda1d1"} Nov 03 00:15:01.080431 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.080405 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589"] Nov 03 00:15:01.082892 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:15:01.082854 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82d527a_412c_4e3d_a8fe_afc3557f9ee5.slice/crio-26dbdfcb7ce1e3ff42a66a8dcdbbf92a2a0e68eb0aeb6f09b525c46dbd9277bb WatchSource:0}: Error finding container 26dbdfcb7ce1e3ff42a66a8dcdbbf92a2a0e68eb0aeb6f09b525c46dbd9277bb: Status 404 returned error can't find the container with id 26dbdfcb7ce1e3ff42a66a8dcdbbf92a2a0e68eb0aeb6f09b525c46dbd9277bb Nov 03 00:15:01.088918 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.088573 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc"] Nov 03 00:15:01.090898 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:15:01.090870 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf186952f_5082_4cf7_bad8_a6ae7c4089b5.slice/crio-704a8a46c8fcdd88c0dfbbbc1d7be2ab7a77a950558d94a024a1b4b6bd85d0a2 WatchSource:0}: Error finding container 704a8a46c8fcdd88c0dfbbbc1d7be2ab7a77a950558d94a024a1b4b6bd85d0a2: Status 404 returned error can't find the container with id 704a8a46c8fcdd88c0dfbbbc1d7be2ab7a77a950558d94a024a1b4b6bd85d0a2 Nov 03 00:15:01.091879 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.091858 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m"] Nov 03 00:15:01.096517 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:15:01.096491 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7056ae18_f4aa_4cb5_b53b_70bd53e7c477.slice/crio-669b46e80d85467ea964f71829876a379e8be1d1d884ed01427cf345fd83bf6f WatchSource:0}: Error finding container 669b46e80d85467ea964f71829876a379e8be1d1d884ed01427cf345fd83bf6f: Status 404 returned error can't find the container with id 669b46e80d85467ea964f71829876a379e8be1d1d884ed01427cf345fd83bf6f Nov 03 00:15:01.128354 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.128106 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg"] Nov 03 00:15:01.130874 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:15:01.130844 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59278e2_911a_426e_80dc_7ed972e6fd36.slice/crio-9282ee515f9a77556424dc904b506e0d9a9116808af54bda70d09388a76fdba5 WatchSource:0}: Error finding container 9282ee515f9a77556424dc904b506e0d9a9116808af54bda70d09388a76fdba5: Status 404 returned error can't find the container with id 9282ee515f9a77556424dc904b506e0d9a9116808af54bda70d09388a76fdba5 Nov 03 00:15:01.138384 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.138245 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8"] Nov 03 00:15:01.152846 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.152820 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g"] Nov 03 00:15:01.182276 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.182229 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" podStartSLOduration=2.1822149299999998 podStartE2EDuration="2.18221493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:01.179185129 +0000 UTC m=+4894.586496962" watchObservedRunningTime="2026-11-03 00:15:01.18221493 +0000 UTC m=+4894.589526761" Nov 03 00:15:01.239952 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.239927 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts"] Nov 03 00:15:01.260129 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.260100 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp"] Nov 03 00:15:01.262344 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:15:01.262309 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb684d2_d193_4bf7_a07d_4e9c1d4116b2.slice/crio-a456eafa06dd772568a815931e76145de7966680e86dc85dd2031eb3148ac3a7 WatchSource:0}: Error finding container a456eafa06dd772568a815931e76145de7966680e86dc85dd2031eb3148ac3a7: Status 404 returned error can't find the container with id a456eafa06dd772568a815931e76145de7966680e86dc85dd2031eb3148ac3a7 Nov 03 00:15:01.730288 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.730064 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" podUID="3941edda-d840-46e5-b257-d42202cd8214" containerName="payload-processing" containerID="cri-o://e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee" gracePeriod=25 Nov 03 00:15:01.949375 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.949288 5672 generic.go:356] "Generic (PLEG): container finished" podID="2144ab98-3fe5-4849-b80c-507d4e015a46" containerID="867ca72a64b95bf3a2e18161a648f1f76a93fef47986594d79618faace5be2cf" exitCode=0 Nov 03 00:15:01.949749 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.949374 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" event={"ID":"2144ab98-3fe5-4849-b80c-507d4e015a46","Type":"ContainerDied","Data":"867ca72a64b95bf3a2e18161a648f1f76a93fef47986594d79618faace5be2cf"} Nov 03 00:15:01.951857 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.951830 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" event={"ID":"7056ae18-f4aa-4cb5-b53b-70bd53e7c477","Type":"ContainerStarted","Data":"cf90810a9103abb1d7324c5124f49164be6e05ef31db5107e5835ae5cbc11b7d"} Nov 03 00:15:01.951992 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.951883 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" event={"ID":"7056ae18-f4aa-4cb5-b53b-70bd53e7c477","Type":"ContainerStarted","Data":"669b46e80d85467ea964f71829876a379e8be1d1d884ed01427cf345fd83bf6f"} Nov 03 00:15:01.960253 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.960220 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" event={"ID":"3c80c505-2325-46ed-bdb0-95899c8ec5d8","Type":"ContainerStarted","Data":"659c1b7d76c4a0a0986bba75744b30afbfbf153f9192055175ffddb1cc9ac85f"} Nov 03 00:15:01.960374 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.960273 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" event={"ID":"3c80c505-2325-46ed-bdb0-95899c8ec5d8","Type":"ContainerStarted","Data":"db6d51899fb27c81a182122c734dd292a3dee4191fdb4c7f278fcd78012d48e7"} Nov 03 00:15:01.961899 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.961869 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" event={"ID":"d59278e2-911a-426e-80dc-7ed972e6fd36","Type":"ContainerStarted","Data":"dc33faf7c215f20c9649fea2bf76d25cf3d89f020b89d47c9f3fc34331df677e"} Nov 03 00:15:01.962020 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.961925 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" event={"ID":"d59278e2-911a-426e-80dc-7ed972e6fd36","Type":"ContainerStarted","Data":"9282ee515f9a77556424dc904b506e0d9a9116808af54bda70d09388a76fdba5"} Nov 03 00:15:01.963654 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.963630 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" event={"ID":"583aad32-dafe-46d4-861f-f07566330db8","Type":"ContainerStarted","Data":"7faccab4faa9e9e97bd0ecf30ad6df84b9a91c181a61d013d311fa46d62f672b"} Nov 03 00:15:01.963769 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.963674 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" event={"ID":"583aad32-dafe-46d4-861f-f07566330db8","Type":"ContainerStarted","Data":"96031bc066fbae310ff1183f1808f43a0ddcce37eb3e43c14da5c4dab4c2853a"} Nov 03 00:15:01.965479 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.965454 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" event={"ID":"d82d527a-412c-4e3d-a8fe-afc3557f9ee5","Type":"ContainerStarted","Data":"b503190cbf94d6d7ce8c8c4a8e0b6da7badb2cb015ceabd54bf7a7092a98f5a0"} Nov 03 00:15:01.965582 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.965497 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" event={"ID":"d82d527a-412c-4e3d-a8fe-afc3557f9ee5","Type":"ContainerStarted","Data":"26dbdfcb7ce1e3ff42a66a8dcdbbf92a2a0e68eb0aeb6f09b525c46dbd9277bb"} Nov 03 00:15:01.967093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.967061 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" event={"ID":"f186952f-5082-4cf7-bad8-a6ae7c4089b5","Type":"ContainerStarted","Data":"cef443f14a3e99666665bea2a687f018ba32e233ab08e726e7e1ffe20f0cc27c"} Nov 03 00:15:01.967229 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.967115 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" event={"ID":"f186952f-5082-4cf7-bad8-a6ae7c4089b5","Type":"ContainerStarted","Data":"704a8a46c8fcdd88c0dfbbbc1d7be2ab7a77a950558d94a024a1b4b6bd85d0a2"} Nov 03 00:15:01.968820 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.968800 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" event={"ID":"74c90ca2-20e4-4c85-94de-9e7127186ea9","Type":"ContainerStarted","Data":"74d3ee698777b2699cde484ba09b9eec99b547357296ced08973dcca60816ad9"} Nov 03 00:15:01.968930 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.968846 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" event={"ID":"74c90ca2-20e4-4c85-94de-9e7127186ea9","Type":"ContainerStarted","Data":"84c0fd8efd6b175d0f32ea417d957ec0e827bea1194b2ac2e7da76362e2f874d"} Nov 03 00:15:01.970446 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.970424 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" event={"ID":"acb684d2-d193-4bf7-a07d-4e9c1d4116b2","Type":"ContainerStarted","Data":"c114834c1dcb24be5ae11634c54a8de7872e2d06c6f37894a5af3920100fe5f1"} Nov 03 00:15:01.970544 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:01.970466 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" event={"ID":"acb684d2-d193-4bf7-a07d-4e9c1d4116b2","Type":"ContainerStarted","Data":"a456eafa06dd772568a815931e76145de7966680e86dc85dd2031eb3148ac3a7"} Nov 03 00:15:02.162089 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.162027 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" podStartSLOduration=2.16200929 podStartE2EDuration="2.16200929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:02.157459135 +0000 UTC m=+4895.564770965" watchObservedRunningTime="2026-11-03 00:15:02.16200929 +0000 UTC m=+4895.569321121" Nov 03 00:15:02.267415 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.262847 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" podStartSLOduration=2.262829633 podStartE2EDuration="2.262829633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:02.26184119 +0000 UTC m=+4895.669153021" watchObservedRunningTime="2026-11-03 00:15:02.262829633 +0000 UTC m=+4895.670141465" Nov 03 00:15:02.368535 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.368473 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" podStartSLOduration=2.368456505 podStartE2EDuration="2.368456505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:02.366338897 +0000 UTC m=+4895.773650730" watchObservedRunningTime="2026-11-03 00:15:02.368456505 +0000 UTC m=+4895.775768337" Nov 03 00:15:02.472624 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.472564 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" podStartSLOduration=2.472543453 podStartE2EDuration="2.472543453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:02.470478933 +0000 UTC m=+4895.877790764" watchObservedRunningTime="2026-11-03 00:15:02.472543453 +0000 UTC m=+4895.879855290" Nov 03 00:15:02.575589 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.575533 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" podStartSLOduration=2.575518882 podStartE2EDuration="2.575518882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:02.574478414 +0000 UTC m=+4895.981790246" watchObservedRunningTime="2026-11-03 00:15:02.575518882 +0000 UTC m=+4895.982830722" Nov 03 00:15:02.681984 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.681931 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" podStartSLOduration=2.681919073 podStartE2EDuration="2.681919073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:02.680601686 +0000 UTC m=+4896.087913517" watchObservedRunningTime="2026-11-03 00:15:02.681919073 +0000 UTC m=+4896.089230902" Nov 03 00:15:02.790124 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.790026 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" podStartSLOduration=2.790009973 podStartE2EDuration="2.790009973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:02.788004559 +0000 UTC m=+4896.195316390" watchObservedRunningTime="2026-11-03 00:15:02.790009973 +0000 UTC m=+4896.197321802" Nov 03 00:15:02.894684 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.894634 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" podStartSLOduration=2.894619921 podStartE2EDuration="2.894619921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:02.893186568 +0000 UTC m=+4896.300498395" watchObservedRunningTime="2026-11-03 00:15:02.894619921 +0000 UTC m=+4896.301931751" Nov 03 00:15:02.980100 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.980068 5672 generic.go:356] "Generic (PLEG): container finished" podID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerID="dc33faf7c215f20c9649fea2bf76d25cf3d89f020b89d47c9f3fc34331df677e" exitCode=60 Nov 03 00:15:02.980575 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.980178 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" event={"ID":"d59278e2-911a-426e-80dc-7ed972e6fd36","Type":"ContainerDied","Data":"dc33faf7c215f20c9649fea2bf76d25cf3d89f020b89d47c9f3fc34331df677e"} Nov 03 00:15:02.980575 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.980392 5672 scope.go:122] "RemoveContainer" containerID="dc33faf7c215f20c9649fea2bf76d25cf3d89f020b89d47c9f3fc34331df677e" Nov 03 00:15:02.981742 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.981722 5672 generic.go:356] "Generic (PLEG): container finished" podID="583aad32-dafe-46d4-861f-f07566330db8" containerID="7faccab4faa9e9e97bd0ecf30ad6df84b9a91c181a61d013d311fa46d62f672b" exitCode=60 Nov 03 00:15:02.981832 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.981790 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" event={"ID":"583aad32-dafe-46d4-861f-f07566330db8","Type":"ContainerDied","Data":"7faccab4faa9e9e97bd0ecf30ad6df84b9a91c181a61d013d311fa46d62f672b"} Nov 03 00:15:02.981975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.981960 5672 scope.go:122] "RemoveContainer" containerID="7faccab4faa9e9e97bd0ecf30ad6df84b9a91c181a61d013d311fa46d62f672b" Nov 03 00:15:02.983502 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.983483 5672 generic.go:356] "Generic (PLEG): container finished" podID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerID="b503190cbf94d6d7ce8c8c4a8e0b6da7badb2cb015ceabd54bf7a7092a98f5a0" exitCode=60 Nov 03 00:15:02.983780 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.983587 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" event={"ID":"d82d527a-412c-4e3d-a8fe-afc3557f9ee5","Type":"ContainerDied","Data":"b503190cbf94d6d7ce8c8c4a8e0b6da7badb2cb015ceabd54bf7a7092a98f5a0"} Nov 03 00:15:02.983842 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.983808 5672 scope.go:122] "RemoveContainer" containerID="b503190cbf94d6d7ce8c8c4a8e0b6da7badb2cb015ceabd54bf7a7092a98f5a0" Nov 03 00:15:02.985295 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.985276 5672 generic.go:356] "Generic (PLEG): container finished" podID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerID="cef443f14a3e99666665bea2a687f018ba32e233ab08e726e7e1ffe20f0cc27c" exitCode=60 Nov 03 00:15:02.985392 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.985334 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" event={"ID":"f186952f-5082-4cf7-bad8-a6ae7c4089b5","Type":"ContainerDied","Data":"cef443f14a3e99666665bea2a687f018ba32e233ab08e726e7e1ffe20f0cc27c"} Nov 03 00:15:02.985529 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.985512 5672 scope.go:122] "RemoveContainer" containerID="cef443f14a3e99666665bea2a687f018ba32e233ab08e726e7e1ffe20f0cc27c" Nov 03 00:15:02.986939 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.986917 5672 generic.go:356] "Generic (PLEG): container finished" podID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerID="74d3ee698777b2699cde484ba09b9eec99b547357296ced08973dcca60816ad9" exitCode=60 Nov 03 00:15:02.987042 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.987000 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" event={"ID":"74c90ca2-20e4-4c85-94de-9e7127186ea9","Type":"ContainerDied","Data":"74d3ee698777b2699cde484ba09b9eec99b547357296ced08973dcca60816ad9"} Nov 03 00:15:02.987211 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.987196 5672 scope.go:122] "RemoveContainer" containerID="74d3ee698777b2699cde484ba09b9eec99b547357296ced08973dcca60816ad9" Nov 03 00:15:02.988358 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.988340 5672 generic.go:356] "Generic (PLEG): container finished" podID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerID="c114834c1dcb24be5ae11634c54a8de7872e2d06c6f37894a5af3920100fe5f1" exitCode=60 Nov 03 00:15:02.988446 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.988421 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" event={"ID":"acb684d2-d193-4bf7-a07d-4e9c1d4116b2","Type":"ContainerDied","Data":"c114834c1dcb24be5ae11634c54a8de7872e2d06c6f37894a5af3920100fe5f1"} Nov 03 00:15:02.988552 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.988535 5672 scope.go:122] "RemoveContainer" containerID="c114834c1dcb24be5ae11634c54a8de7872e2d06c6f37894a5af3920100fe5f1" Nov 03 00:15:02.990067 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.990046 5672 generic.go:356] "Generic (PLEG): container finished" podID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerID="cf90810a9103abb1d7324c5124f49164be6e05ef31db5107e5835ae5cbc11b7d" exitCode=60 Nov 03 00:15:02.990171 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.990125 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" event={"ID":"7056ae18-f4aa-4cb5-b53b-70bd53e7c477","Type":"ContainerDied","Data":"cf90810a9103abb1d7324c5124f49164be6e05ef31db5107e5835ae5cbc11b7d"} Nov 03 00:15:02.990315 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.990299 5672 scope.go:122] "RemoveContainer" containerID="cf90810a9103abb1d7324c5124f49164be6e05ef31db5107e5835ae5cbc11b7d" Nov 03 00:15:02.992668 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.992649 5672 generic.go:356] "Generic (PLEG): container finished" podID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerID="659c1b7d76c4a0a0986bba75744b30afbfbf153f9192055175ffddb1cc9ac85f" exitCode=60 Nov 03 00:15:02.992771 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.992734 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" event={"ID":"3c80c505-2325-46ed-bdb0-95899c8ec5d8","Type":"ContainerDied","Data":"659c1b7d76c4a0a0986bba75744b30afbfbf153f9192055175ffddb1cc9ac85f"} Nov 03 00:15:02.992904 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:02.992889 5672 scope.go:122] "RemoveContainer" containerID="659c1b7d76c4a0a0986bba75744b30afbfbf153f9192055175ffddb1cc9ac85f" Nov 03 00:15:03.480797 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:03.480769 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:15:03.629366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:03.629335 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-xks-ca\") pod \"2144ab98-3fe5-4849-b80c-507d4e015a46\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " Nov 03 00:15:03.629541 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:03.629403 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2144ab98-3fe5-4849-b80c-507d4e015a46-kube-api-access-6cwtt\" (UniqueName: \"kubernetes.io/projected/2144ab98-3fe5-4849-b80c-507d4e015a46-kube-api-access-6cwtt\") pod \"2144ab98-3fe5-4849-b80c-507d4e015a46\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " Nov 03 00:15:03.629541 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:03.629448 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-openshift-ca\") pod \"2144ab98-3fe5-4849-b80c-507d4e015a46\" (UID: \"2144ab98-3fe5-4849-b80c-507d4e015a46\") " Nov 03 00:15:03.630229 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:03.630202 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-openshift-ca" pod "2144ab98-3fe5-4849-b80c-507d4e015a46" (UID: "2144ab98-3fe5-4849-b80c-507d4e015a46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:03.633596 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:03.633554 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2144ab98-3fe5-4849-b80c-507d4e015a46-kube-api-access-6cwtt" pod "2144ab98-3fe5-4849-b80c-507d4e015a46" (UID: "2144ab98-3fe5-4849-b80c-507d4e015a46"). InnerVolumeSpecName "kube-api-access-6cwtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:03.634494 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:03.634452 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-xks-ca" pod "2144ab98-3fe5-4849-b80c-507d4e015a46" (UID: "2144ab98-3fe5-4849-b80c-507d4e015a46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:03.731763 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:03.731731 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:03.731959 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:03.731769 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cwtt\" (UniqueName: \"kubernetes.io/projected/2144ab98-3fe5-4849-b80c-507d4e015a46-kube-api-access-6cwtt\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:03.731959 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:03.731785 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2144ab98-3fe5-4849-b80c-507d4e015a46-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:04.001878 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:04.001797 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" event={"ID":"7056ae18-f4aa-4cb5-b53b-70bd53e7c477","Type":"ContainerStarted","Data":"7cfc63c34c6aa519770e8b8bc8d1daaa43c0ad4c59547b4734827141d2069c79"} Nov 03 00:15:04.004002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:04.003971 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" event={"ID":"3c80c505-2325-46ed-bdb0-95899c8ec5d8","Type":"ContainerStarted","Data":"a24dc2dd1321aed4664f84ac94eae80b37f5113a39dffc4cb069374a6fa786b5"} Nov 03 00:15:04.006099 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:04.006073 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" event={"ID":"d59278e2-911a-426e-80dc-7ed972e6fd36","Type":"ContainerStarted","Data":"14cca7a51a18a43aa0f0f4b3cac0d2e9d776182162b1e9505b1273782da7baec"} Nov 03 00:15:04.008210 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:04.008184 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" event={"ID":"583aad32-dafe-46d4-861f-f07566330db8","Type":"ContainerStarted","Data":"53f279b6a1468c153861ee9639a5bc09f97a2fc672042ef118086c96d2ef663b"} Nov 03 00:15:04.010198 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:04.010177 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" event={"ID":"d82d527a-412c-4e3d-a8fe-afc3557f9ee5","Type":"ContainerStarted","Data":"dbab5a98bc5249fc923c26bad8c99f348b6b62dc1d3e920539188053a3ff4b22"} Nov 03 00:15:04.011950 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:04.011926 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" event={"ID":"f186952f-5082-4cf7-bad8-a6ae7c4089b5","Type":"ContainerStarted","Data":"c10f05aff5975494e4da064804b27265e960ade617ca1c02497422cc202dfd39"} Nov 03 00:15:04.013743 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:04.013722 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" event={"ID":"74c90ca2-20e4-4c85-94de-9e7127186ea9","Type":"ContainerStarted","Data":"e2f2a7638179aac2a0e7bd0c95b2f7b5c131eeee00a0d81a559fb4b16b819188"} Nov 03 00:15:04.015514 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:04.015492 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" event={"ID":"acb684d2-d193-4bf7-a07d-4e9c1d4116b2","Type":"ContainerStarted","Data":"27bb42ff4f035566616a6025969e3cd228fd47969015d037284935f28ae8e984"} Nov 03 00:15:04.016753 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:04.016736 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" event={"ID":"2144ab98-3fe5-4849-b80c-507d4e015a46","Type":"ContainerDied","Data":"05cc4ede470d0be84d724bacc5bbb654fa72326d1a62c3deb5e569c309bda1d1"} Nov 03 00:15:04.016825 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:04.016764 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6" Nov 03 00:15:04.016865 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:04.016768 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05cc4ede470d0be84d724bacc5bbb654fa72326d1a62c3deb5e569c309bda1d1" Nov 03 00:15:05.025395 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.025359 5672 generic.go:356] "Generic (PLEG): container finished" podID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerID="14cca7a51a18a43aa0f0f4b3cac0d2e9d776182162b1e9505b1273782da7baec" exitCode=60 Nov 03 00:15:05.025854 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.025440 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" event={"ID":"d59278e2-911a-426e-80dc-7ed972e6fd36","Type":"ContainerDied","Data":"14cca7a51a18a43aa0f0f4b3cac0d2e9d776182162b1e9505b1273782da7baec"} Nov 03 00:15:05.025854 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.025510 5672 scope.go:122] "RemoveContainer" containerID="dc33faf7c215f20c9649fea2bf76d25cf3d89f020b89d47c9f3fc34331df677e" Nov 03 00:15:05.025854 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.025807 5672 scope.go:122] "RemoveContainer" containerID="14cca7a51a18a43aa0f0f4b3cac0d2e9d776182162b1e9505b1273782da7baec" Nov 03 00:15:05.026210 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:05.026182 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-hpvgg_odh-ai-gateway-infra(d59278e2-911a-426e-80dc-7ed972e6fd36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" Nov 03 00:15:05.028047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.028028 5672 generic.go:356] "Generic (PLEG): container finished" podID="583aad32-dafe-46d4-861f-f07566330db8" containerID="53f279b6a1468c153861ee9639a5bc09f97a2fc672042ef118086c96d2ef663b" exitCode=60 Nov 03 00:15:05.028208 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.028101 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" event={"ID":"583aad32-dafe-46d4-861f-f07566330db8","Type":"ContainerDied","Data":"53f279b6a1468c153861ee9639a5bc09f97a2fc672042ef118086c96d2ef663b"} Nov 03 00:15:05.028387 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.028373 5672 scope.go:122] "RemoveContainer" containerID="53f279b6a1468c153861ee9639a5bc09f97a2fc672042ef118086c96d2ef663b" Nov 03 00:15:05.028658 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:05.028637 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8_odh-ai-gateway-infra(583aad32-dafe-46d4-861f-f07566330db8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" podUID="583aad32-dafe-46d4-861f-f07566330db8" Nov 03 00:15:05.030087 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.030066 5672 generic.go:356] "Generic (PLEG): container finished" podID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerID="dbab5a98bc5249fc923c26bad8c99f348b6b62dc1d3e920539188053a3ff4b22" exitCode=60 Nov 03 00:15:05.030219 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.030144 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" event={"ID":"d82d527a-412c-4e3d-a8fe-afc3557f9ee5","Type":"ContainerDied","Data":"dbab5a98bc5249fc923c26bad8c99f348b6b62dc1d3e920539188053a3ff4b22"} Nov 03 00:15:05.030484 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.030469 5672 scope.go:122] "RemoveContainer" containerID="dbab5a98bc5249fc923c26bad8c99f348b6b62dc1d3e920539188053a3ff4b22" Nov 03 00:15:05.030729 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:05.030706 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589_odh-ai-gateway-infra(d82d527a-412c-4e3d-a8fe-afc3557f9ee5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" Nov 03 00:15:05.032593 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.032566 5672 generic.go:356] "Generic (PLEG): container finished" podID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerID="c10f05aff5975494e4da064804b27265e960ade617ca1c02497422cc202dfd39" exitCode=60 Nov 03 00:15:05.032707 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.032648 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" event={"ID":"f186952f-5082-4cf7-bad8-a6ae7c4089b5","Type":"ContainerDied","Data":"c10f05aff5975494e4da064804b27265e960ade617ca1c02497422cc202dfd39"} Nov 03 00:15:05.032833 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.032818 5672 scope.go:122] "RemoveContainer" containerID="c10f05aff5975494e4da064804b27265e960ade617ca1c02497422cc202dfd39" Nov 03 00:15:05.033067 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:05.033043 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc_odh-ai-gateway-infra(f186952f-5082-4cf7-bad8-a6ae7c4089b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" Nov 03 00:15:05.034901 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.034879 5672 generic.go:356] "Generic (PLEG): container finished" podID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerID="e2f2a7638179aac2a0e7bd0c95b2f7b5c131eeee00a0d81a559fb4b16b819188" exitCode=60 Nov 03 00:15:05.035001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.034953 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" event={"ID":"74c90ca2-20e4-4c85-94de-9e7127186ea9","Type":"ContainerDied","Data":"e2f2a7638179aac2a0e7bd0c95b2f7b5c131eeee00a0d81a559fb4b16b819188"} Nov 03 00:15:05.035134 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.035117 5672 scope.go:122] "RemoveContainer" containerID="e2f2a7638179aac2a0e7bd0c95b2f7b5c131eeee00a0d81a559fb4b16b819188" Nov 03 00:15:05.035331 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:05.035313 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts_odh-ai-gateway-infra(74c90ca2-20e4-4c85-94de-9e7127186ea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" Nov 03 00:15:05.037132 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.037110 5672 generic.go:356] "Generic (PLEG): container finished" podID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerID="27bb42ff4f035566616a6025969e3cd228fd47969015d037284935f28ae8e984" exitCode=60 Nov 03 00:15:05.037264 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.037220 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" event={"ID":"acb684d2-d193-4bf7-a07d-4e9c1d4116b2","Type":"ContainerDied","Data":"27bb42ff4f035566616a6025969e3cd228fd47969015d037284935f28ae8e984"} Nov 03 00:15:05.037390 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.037372 5672 scope.go:122] "RemoveContainer" containerID="27bb42ff4f035566616a6025969e3cd228fd47969015d037284935f28ae8e984" Nov 03 00:15:05.037634 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:05.037616 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp_odh-ai-gateway-infra(acb684d2-d193-4bf7-a07d-4e9c1d4116b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" Nov 03 00:15:05.039637 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.039614 5672 generic.go:356] "Generic (PLEG): container finished" podID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerID="7cfc63c34c6aa519770e8b8bc8d1daaa43c0ad4c59547b4734827141d2069c79" exitCode=60 Nov 03 00:15:05.039751 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.039654 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" event={"ID":"7056ae18-f4aa-4cb5-b53b-70bd53e7c477","Type":"ContainerDied","Data":"7cfc63c34c6aa519770e8b8bc8d1daaa43c0ad4c59547b4734827141d2069c79"} Nov 03 00:15:05.040086 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.040066 5672 scope.go:122] "RemoveContainer" containerID="7cfc63c34c6aa519770e8b8bc8d1daaa43c0ad4c59547b4734827141d2069c79" Nov 03 00:15:05.040343 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:05.040316 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m_odh-ai-gateway-infra(7056ae18-f4aa-4cb5-b53b-70bd53e7c477)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" Nov 03 00:15:05.041322 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.041301 5672 scope.go:122] "RemoveContainer" containerID="7faccab4faa9e9e97bd0ecf30ad6df84b9a91c181a61d013d311fa46d62f672b" Nov 03 00:15:05.041831 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.041811 5672 generic.go:356] "Generic (PLEG): container finished" podID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerID="a24dc2dd1321aed4664f84ac94eae80b37f5113a39dffc4cb069374a6fa786b5" exitCode=60 Nov 03 00:15:05.041943 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.041896 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" event={"ID":"3c80c505-2325-46ed-bdb0-95899c8ec5d8","Type":"ContainerDied","Data":"a24dc2dd1321aed4664f84ac94eae80b37f5113a39dffc4cb069374a6fa786b5"} Nov 03 00:15:05.042115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.042083 5672 scope.go:122] "RemoveContainer" containerID="a24dc2dd1321aed4664f84ac94eae80b37f5113a39dffc4cb069374a6fa786b5" Nov 03 00:15:05.042359 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:05.042337 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g_odh-ai-gateway-infra(3c80c505-2325-46ed-bdb0-95899c8ec5d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" Nov 03 00:15:05.056697 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.056678 5672 scope.go:122] "RemoveContainer" containerID="b503190cbf94d6d7ce8c8c4a8e0b6da7badb2cb015ceabd54bf7a7092a98f5a0" Nov 03 00:15:05.071663 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.071644 5672 scope.go:122] "RemoveContainer" containerID="cef443f14a3e99666665bea2a687f018ba32e233ab08e726e7e1ffe20f0cc27c" Nov 03 00:15:05.084457 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.084438 5672 scope.go:122] "RemoveContainer" containerID="74d3ee698777b2699cde484ba09b9eec99b547357296ced08973dcca60816ad9" Nov 03 00:15:05.096644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.096628 5672 scope.go:122] "RemoveContainer" containerID="c114834c1dcb24be5ae11634c54a8de7872e2d06c6f37894a5af3920100fe5f1" Nov 03 00:15:05.110006 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.109990 5672 scope.go:122] "RemoveContainer" containerID="cf90810a9103abb1d7324c5124f49164be6e05ef31db5107e5835ae5cbc11b7d" Nov 03 00:15:05.121906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:05.121889 5672 scope.go:122] "RemoveContainer" containerID="659c1b7d76c4a0a0986bba75744b30afbfbf153f9192055175ffddb1cc9ac85f" Nov 03 00:15:06.052262 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:06.052235 5672 scope.go:122] "RemoveContainer" containerID="27bb42ff4f035566616a6025969e3cd228fd47969015d037284935f28ae8e984" Nov 03 00:15:06.052754 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:06.052546 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp_odh-ai-gateway-infra(acb684d2-d193-4bf7-a07d-4e9c1d4116b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" Nov 03 00:15:06.054158 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:06.054121 5672 scope.go:122] "RemoveContainer" containerID="7cfc63c34c6aa519770e8b8bc8d1daaa43c0ad4c59547b4734827141d2069c79" Nov 03 00:15:06.054475 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:06.054445 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m_odh-ai-gateway-infra(7056ae18-f4aa-4cb5-b53b-70bd53e7c477)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" Nov 03 00:15:06.055611 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:06.055587 5672 scope.go:122] "RemoveContainer" containerID="a24dc2dd1321aed4664f84ac94eae80b37f5113a39dffc4cb069374a6fa786b5" Nov 03 00:15:06.055849 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:06.055817 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g_odh-ai-gateway-infra(3c80c505-2325-46ed-bdb0-95899c8ec5d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" Nov 03 00:15:06.057430 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:06.057411 5672 scope.go:122] "RemoveContainer" containerID="14cca7a51a18a43aa0f0f4b3cac0d2e9d776182162b1e9505b1273782da7baec" Nov 03 00:15:06.057655 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:06.057635 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-hpvgg_odh-ai-gateway-infra(d59278e2-911a-426e-80dc-7ed972e6fd36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" Nov 03 00:15:06.059044 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:06.059027 5672 scope.go:122] "RemoveContainer" containerID="53f279b6a1468c153861ee9639a5bc09f97a2fc672042ef118086c96d2ef663b" Nov 03 00:15:06.059250 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:06.059230 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8_odh-ai-gateway-infra(583aad32-dafe-46d4-861f-f07566330db8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" podUID="583aad32-dafe-46d4-861f-f07566330db8" Nov 03 00:15:06.060626 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:06.060610 5672 scope.go:122] "RemoveContainer" containerID="dbab5a98bc5249fc923c26bad8c99f348b6b62dc1d3e920539188053a3ff4b22" Nov 03 00:15:06.060801 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:06.060780 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589_odh-ai-gateway-infra(d82d527a-412c-4e3d-a8fe-afc3557f9ee5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" Nov 03 00:15:06.062272 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:06.062251 5672 scope.go:122] "RemoveContainer" containerID="c10f05aff5975494e4da064804b27265e960ade617ca1c02497422cc202dfd39" Nov 03 00:15:06.062475 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:06.062453 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc_odh-ai-gateway-infra(f186952f-5082-4cf7-bad8-a6ae7c4089b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" Nov 03 00:15:06.063877 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:06.063858 5672 scope.go:122] "RemoveContainer" containerID="e2f2a7638179aac2a0e7bd0c95b2f7b5c131eeee00a0d81a559fb4b16b819188" Nov 03 00:15:06.064077 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:06.064056 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts_odh-ai-gateway-infra(74c90ca2-20e4-4c85-94de-9e7127186ea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" Nov 03 00:15:10.714101 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:10.714056 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" podUID="3941edda-d840-46e5-b257-d42202cd8214" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:15:15.706317 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:15.706279 5672 scope.go:122] "RemoveContainer" containerID="53f279b6a1468c153861ee9639a5bc09f97a2fc672042ef118086c96d2ef663b" Nov 03 00:15:15.706777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:15.706354 5672 scope.go:122] "RemoveContainer" containerID="dbab5a98bc5249fc923c26bad8c99f348b6b62dc1d3e920539188053a3ff4b22" Nov 03 00:15:16.140974 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:16.140940 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" event={"ID":"583aad32-dafe-46d4-861f-f07566330db8","Type":"ContainerStarted","Data":"f664798b48801973d988d65138594851d27211b22fa6ca32742ec9e54a995095"} Nov 03 00:15:16.143109 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:16.143077 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" event={"ID":"d82d527a-412c-4e3d-a8fe-afc3557f9ee5","Type":"ContainerStarted","Data":"194474bced49e677d893789a9cf0eed20c4a0ce44e8a39656cc4959b66c046ef"} Nov 03 00:15:16.709971 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:16.709942 5672 scope.go:122] "RemoveContainer" containerID="a24dc2dd1321aed4664f84ac94eae80b37f5113a39dffc4cb069374a6fa786b5" Nov 03 00:15:16.710434 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:16.710053 5672 scope.go:122] "RemoveContainer" containerID="27bb42ff4f035566616a6025969e3cd228fd47969015d037284935f28ae8e984" Nov 03 00:15:17.151926 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.151896 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" event={"ID":"acb684d2-d193-4bf7-a07d-4e9c1d4116b2","Type":"ContainerStarted","Data":"0d38ce3c24e14ee52fcec5e21bc56356469114d7beb00270c3aa8e2f5fb30e8c"} Nov 03 00:15:17.159617 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.159592 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" event={"ID":"3c80c505-2325-46ed-bdb0-95899c8ec5d8","Type":"ContainerStarted","Data":"deaf4c2ed986bee35a78d281d15796ad9d4e9b50e30b599c91ea42e86ea6e350"} Nov 03 00:15:17.161830 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.161807 5672 generic.go:356] "Generic (PLEG): container finished" podID="583aad32-dafe-46d4-861f-f07566330db8" containerID="f664798b48801973d988d65138594851d27211b22fa6ca32742ec9e54a995095" exitCode=60 Nov 03 00:15:17.161950 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.161877 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" event={"ID":"583aad32-dafe-46d4-861f-f07566330db8","Type":"ContainerDied","Data":"f664798b48801973d988d65138594851d27211b22fa6ca32742ec9e54a995095"} Nov 03 00:15:17.161950 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.161938 5672 scope.go:122] "RemoveContainer" containerID="53f279b6a1468c153861ee9639a5bc09f97a2fc672042ef118086c96d2ef663b" Nov 03 00:15:17.162311 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.162294 5672 scope.go:122] "RemoveContainer" containerID="f664798b48801973d988d65138594851d27211b22fa6ca32742ec9e54a995095" Nov 03 00:15:17.162645 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:17.162610 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8_odh-ai-gateway-infra(583aad32-dafe-46d4-861f-f07566330db8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" podUID="583aad32-dafe-46d4-861f-f07566330db8" Nov 03 00:15:17.164238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.164214 5672 generic.go:356] "Generic (PLEG): container finished" podID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerID="194474bced49e677d893789a9cf0eed20c4a0ce44e8a39656cc4959b66c046ef" exitCode=60 Nov 03 00:15:17.164341 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.164288 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" event={"ID":"d82d527a-412c-4e3d-a8fe-afc3557f9ee5","Type":"ContainerDied","Data":"194474bced49e677d893789a9cf0eed20c4a0ce44e8a39656cc4959b66c046ef"} Nov 03 00:15:17.164602 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.164589 5672 scope.go:122] "RemoveContainer" containerID="194474bced49e677d893789a9cf0eed20c4a0ce44e8a39656cc4959b66c046ef" Nov 03 00:15:17.164827 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:17.164809 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589_odh-ai-gateway-infra(d82d527a-412c-4e3d-a8fe-afc3557f9ee5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" Nov 03 00:15:17.173874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.173854 5672 scope.go:122] "RemoveContainer" containerID="dbab5a98bc5249fc923c26bad8c99f348b6b62dc1d3e920539188053a3ff4b22" Nov 03 00:15:17.274877 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.274839 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8"] Nov 03 00:15:17.374538 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.374499 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589"] Nov 03 00:15:17.705588 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.705554 5672 scope.go:122] "RemoveContainer" containerID="e2f2a7638179aac2a0e7bd0c95b2f7b5c131eeee00a0d81a559fb4b16b819188" Nov 03 00:15:17.705851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.705828 5672 scope.go:122] "RemoveContainer" containerID="c10f05aff5975494e4da064804b27265e960ade617ca1c02497422cc202dfd39" Nov 03 00:15:17.705945 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:17.705927 5672 scope.go:122] "RemoveContainer" containerID="14cca7a51a18a43aa0f0f4b3cac0d2e9d776182162b1e9505b1273782da7baec" Nov 03 00:15:18.173772 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.173742 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" event={"ID":"d59278e2-911a-426e-80dc-7ed972e6fd36","Type":"ContainerStarted","Data":"a419402e084bba5442b707df634d49072068940512e6b1bc0978a798161fa8ec"} Nov 03 00:15:18.190192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.180272 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" event={"ID":"f186952f-5082-4cf7-bad8-a6ae7c4089b5","Type":"ContainerStarted","Data":"a787d6c5faca75f39bd8d1719d47c627eb4ef16c154abc6b8a3391f08605643d"} Nov 03 00:15:18.190192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.182523 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" event={"ID":"74c90ca2-20e4-4c85-94de-9e7127186ea9","Type":"ContainerStarted","Data":"41bf3b000a24bf743a09bcf75607556738317117f3da6c87efc9e908cba0f8c8"} Nov 03 00:15:18.190192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.184938 5672 generic.go:356] "Generic (PLEG): container finished" podID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerID="0d38ce3c24e14ee52fcec5e21bc56356469114d7beb00270c3aa8e2f5fb30e8c" exitCode=60 Nov 03 00:15:18.190192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.184968 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" event={"ID":"acb684d2-d193-4bf7-a07d-4e9c1d4116b2","Type":"ContainerDied","Data":"0d38ce3c24e14ee52fcec5e21bc56356469114d7beb00270c3aa8e2f5fb30e8c"} Nov 03 00:15:18.190192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.185022 5672 scope.go:122] "RemoveContainer" containerID="27bb42ff4f035566616a6025969e3cd228fd47969015d037284935f28ae8e984" Nov 03 00:15:18.190192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.185401 5672 scope.go:122] "RemoveContainer" containerID="0d38ce3c24e14ee52fcec5e21bc56356469114d7beb00270c3aa8e2f5fb30e8c" Nov 03 00:15:18.190192 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:18.185646 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp_odh-ai-gateway-infra(acb684d2-d193-4bf7-a07d-4e9c1d4116b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" Nov 03 00:15:18.190192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.187472 5672 generic.go:356] "Generic (PLEG): container finished" podID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerID="deaf4c2ed986bee35a78d281d15796ad9d4e9b50e30b599c91ea42e86ea6e350" exitCode=60 Nov 03 00:15:18.190192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.187531 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" event={"ID":"3c80c505-2325-46ed-bdb0-95899c8ec5d8","Type":"ContainerDied","Data":"deaf4c2ed986bee35a78d281d15796ad9d4e9b50e30b599c91ea42e86ea6e350"} Nov 03 00:15:18.190192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.187837 5672 scope.go:122] "RemoveContainer" containerID="deaf4c2ed986bee35a78d281d15796ad9d4e9b50e30b599c91ea42e86ea6e350" Nov 03 00:15:18.190192 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:18.188077 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g_odh-ai-gateway-infra(3c80c505-2325-46ed-bdb0-95899c8ec5d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" Nov 03 00:15:18.201252 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.201228 5672 scope.go:122] "RemoveContainer" containerID="a24dc2dd1321aed4664f84ac94eae80b37f5113a39dffc4cb069374a6fa786b5" Nov 03 00:15:18.297077 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.297019 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp"] Nov 03 00:15:18.368612 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.368586 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" Nov 03 00:15:18.393195 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.393166 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" Nov 03 00:15:18.399285 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.399258 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g"] Nov 03 00:15:18.404130 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.404106 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583aad32-dafe-46d4-861f-f07566330db8-kube-api-access-fwlgv\" (UniqueName: \"kubernetes.io/projected/583aad32-dafe-46d4-861f-f07566330db8-kube-api-access-fwlgv\") pod \"583aad32-dafe-46d4-861f-f07566330db8\" (UID: \"583aad32-dafe-46d4-861f-f07566330db8\") " Nov 03 00:15:18.404262 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.404219 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-kube-api-access-7qtr5\" (UniqueName: \"kubernetes.io/projected/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-kube-api-access-7qtr5\") pod \"d82d527a-412c-4e3d-a8fe-afc3557f9ee5\" (UID: \"d82d527a-412c-4e3d-a8fe-afc3557f9ee5\") " Nov 03 00:15:18.404262 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.404253 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583aad32-dafe-46d4-861f-f07566330db8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/583aad32-dafe-46d4-861f-f07566330db8-maas-api-service-ca\") pod \"583aad32-dafe-46d4-861f-f07566330db8\" (UID: \"583aad32-dafe-46d4-861f-f07566330db8\") " Nov 03 00:15:18.404391 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.404360 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-maas-api-service-ca\") pod \"d82d527a-412c-4e3d-a8fe-afc3557f9ee5\" (UID: \"d82d527a-412c-4e3d-a8fe-afc3557f9ee5\") " Nov 03 00:15:18.404701 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.404681 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583aad32-dafe-46d4-861f-f07566330db8-maas-api-service-ca" pod "583aad32-dafe-46d4-861f-f07566330db8" (UID: "583aad32-dafe-46d4-861f-f07566330db8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:18.404770 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.404708 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-maas-api-service-ca" pod "d82d527a-412c-4e3d-a8fe-afc3557f9ee5" (UID: "d82d527a-412c-4e3d-a8fe-afc3557f9ee5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:18.406312 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.406291 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-kube-api-access-7qtr5" pod "d82d527a-412c-4e3d-a8fe-afc3557f9ee5" (UID: "d82d527a-412c-4e3d-a8fe-afc3557f9ee5"). InnerVolumeSpecName "kube-api-access-7qtr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:18.406423 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.406337 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583aad32-dafe-46d4-861f-f07566330db8-kube-api-access-fwlgv" pod "583aad32-dafe-46d4-861f-f07566330db8" (UID: "583aad32-dafe-46d4-861f-f07566330db8"). InnerVolumeSpecName "kube-api-access-fwlgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:18.505435 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.505402 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:18.505435 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.505431 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwlgv\" (UniqueName: \"kubernetes.io/projected/583aad32-dafe-46d4-861f-f07566330db8-kube-api-access-fwlgv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:18.505435 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.505442 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qtr5\" (UniqueName: \"kubernetes.io/projected/d82d527a-412c-4e3d-a8fe-afc3557f9ee5-kube-api-access-7qtr5\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:18.505685 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.505451 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/583aad32-dafe-46d4-861f-f07566330db8-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:18.706480 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:18.706436 5672 scope.go:122] "RemoveContainer" containerID="7cfc63c34c6aa519770e8b8bc8d1daaa43c0ad4c59547b4734827141d2069c79" Nov 03 00:15:19.195972 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.195939 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" event={"ID":"7056ae18-f4aa-4cb5-b53b-70bd53e7c477","Type":"ContainerStarted","Data":"eb8610ac7ff1f4ffe991c7032f72679ce7649dbe7df1f8cd074147e497a3bdd4"} Nov 03 00:15:19.199863 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.199836 5672 generic.go:356] "Generic (PLEG): container finished" podID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerID="a419402e084bba5442b707df634d49072068940512e6b1bc0978a798161fa8ec" exitCode=60 Nov 03 00:15:19.200032 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.199904 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" event={"ID":"d59278e2-911a-426e-80dc-7ed972e6fd36","Type":"ContainerDied","Data":"a419402e084bba5442b707df634d49072068940512e6b1bc0978a798161fa8ec"} Nov 03 00:15:19.200032 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.199956 5672 scope.go:122] "RemoveContainer" containerID="14cca7a51a18a43aa0f0f4b3cac0d2e9d776182162b1e9505b1273782da7baec" Nov 03 00:15:19.200246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.200228 5672 scope.go:122] "RemoveContainer" containerID="a419402e084bba5442b707df634d49072068940512e6b1bc0978a798161fa8ec" Nov 03 00:15:19.200540 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:19.200516 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-hpvgg_odh-ai-gateway-infra(d59278e2-911a-426e-80dc-7ed972e6fd36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" Nov 03 00:15:19.201431 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.201411 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" Nov 03 00:15:19.201566 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.201407 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8" event={"ID":"583aad32-dafe-46d4-861f-f07566330db8","Type":"ContainerDied","Data":"96031bc066fbae310ff1183f1808f43a0ddcce37eb3e43c14da5c4dab4c2853a"} Nov 03 00:15:19.203049 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.203034 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" Nov 03 00:15:19.203175 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.203083 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589" event={"ID":"d82d527a-412c-4e3d-a8fe-afc3557f9ee5","Type":"ContainerDied","Data":"26dbdfcb7ce1e3ff42a66a8dcdbbf92a2a0e68eb0aeb6f09b525c46dbd9277bb"} Nov 03 00:15:19.206100 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.206069 5672 generic.go:356] "Generic (PLEG): container finished" podID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerID="a787d6c5faca75f39bd8d1719d47c627eb4ef16c154abc6b8a3391f08605643d" exitCode=60 Nov 03 00:15:19.206224 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.206183 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" event={"ID":"f186952f-5082-4cf7-bad8-a6ae7c4089b5","Type":"ContainerDied","Data":"a787d6c5faca75f39bd8d1719d47c627eb4ef16c154abc6b8a3391f08605643d"} Nov 03 00:15:19.206517 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.206493 5672 scope.go:122] "RemoveContainer" containerID="a787d6c5faca75f39bd8d1719d47c627eb4ef16c154abc6b8a3391f08605643d" Nov 03 00:15:19.206757 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:19.206729 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc_odh-ai-gateway-infra(f186952f-5082-4cf7-bad8-a6ae7c4089b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" Nov 03 00:15:19.208899 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.208525 5672 generic.go:356] "Generic (PLEG): container finished" podID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerID="41bf3b000a24bf743a09bcf75607556738317117f3da6c87efc9e908cba0f8c8" exitCode=60 Nov 03 00:15:19.208899 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.208621 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" event={"ID":"74c90ca2-20e4-4c85-94de-9e7127186ea9","Type":"ContainerDied","Data":"41bf3b000a24bf743a09bcf75607556738317117f3da6c87efc9e908cba0f8c8"} Nov 03 00:15:19.209041 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.208947 5672 scope.go:122] "RemoveContainer" containerID="41bf3b000a24bf743a09bcf75607556738317117f3da6c87efc9e908cba0f8c8" Nov 03 00:15:19.209203 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:19.209179 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts_odh-ai-gateway-infra(74c90ca2-20e4-4c85-94de-9e7127186ea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" Nov 03 00:15:19.216402 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.216357 5672 scope.go:122] "RemoveContainer" containerID="f664798b48801973d988d65138594851d27211b22fa6ca32742ec9e54a995095" Nov 03 00:15:19.228279 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.227453 5672 scope.go:122] "RemoveContainer" containerID="194474bced49e677d893789a9cf0eed20c4a0ce44e8a39656cc4959b66c046ef" Nov 03 00:15:19.240249 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.240228 5672 scope.go:122] "RemoveContainer" containerID="c10f05aff5975494e4da064804b27265e960ade617ca1c02497422cc202dfd39" Nov 03 00:15:19.254853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.254827 5672 scope.go:122] "RemoveContainer" containerID="e2f2a7638179aac2a0e7bd0c95b2f7b5c131eeee00a0d81a559fb4b16b819188" Nov 03 00:15:19.295962 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.295441 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg"] Nov 03 00:15:19.376362 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.376333 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" Nov 03 00:15:19.398884 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.398859 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc"] Nov 03 00:15:19.407208 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.407187 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589"] Nov 03 00:15:19.411949 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.411925 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894415-nt589"] Nov 03 00:15:19.415541 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.415500 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c80c505-2325-46ed-bdb0-95899c8ec5d8-kube-api-access-42cnl\" (UniqueName: \"kubernetes.io/projected/3c80c505-2325-46ed-bdb0-95899c8ec5d8-kube-api-access-42cnl\") pod \"3c80c505-2325-46ed-bdb0-95899c8ec5d8\" (UID: \"3c80c505-2325-46ed-bdb0-95899c8ec5d8\") " Nov 03 00:15:19.415651 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.415574 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c80c505-2325-46ed-bdb0-95899c8ec5d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c80c505-2325-46ed-bdb0-95899c8ec5d8-maas-api-service-ca\") pod \"3c80c505-2325-46ed-bdb0-95899c8ec5d8\" (UID: \"3c80c505-2325-46ed-bdb0-95899c8ec5d8\") " Nov 03 00:15:19.416219 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.416191 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c80c505-2325-46ed-bdb0-95899c8ec5d8-maas-api-service-ca" pod "3c80c505-2325-46ed-bdb0-95899c8ec5d8" (UID: "3c80c505-2325-46ed-bdb0-95899c8ec5d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:19.418703 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.418104 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c80c505-2325-46ed-bdb0-95899c8ec5d8-kube-api-access-42cnl" pod "3c80c505-2325-46ed-bdb0-95899c8ec5d8" (UID: "3c80c505-2325-46ed-bdb0-95899c8ec5d8"). InnerVolumeSpecName "kube-api-access-42cnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:19.429037 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.429012 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" Nov 03 00:15:19.516531 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.516494 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-maas-api-service-ca\") pod \"acb684d2-d193-4bf7-a07d-4e9c1d4116b2\" (UID: \"acb684d2-d193-4bf7-a07d-4e9c1d4116b2\") " Nov 03 00:15:19.516705 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.516646 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-kube-api-access-p89tc\" (UniqueName: \"kubernetes.io/projected/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-kube-api-access-p89tc\") pod \"acb684d2-d193-4bf7-a07d-4e9c1d4116b2\" (UID: \"acb684d2-d193-4bf7-a07d-4e9c1d4116b2\") " Nov 03 00:15:19.516915 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.516890 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-maas-api-service-ca" pod "acb684d2-d193-4bf7-a07d-4e9c1d4116b2" (UID: "acb684d2-d193-4bf7-a07d-4e9c1d4116b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:19.517106 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.517087 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:19.517196 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.517112 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42cnl\" (UniqueName: \"kubernetes.io/projected/3c80c505-2325-46ed-bdb0-95899c8ec5d8-kube-api-access-42cnl\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:19.517196 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.517126 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c80c505-2325-46ed-bdb0-95899c8ec5d8-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:19.518726 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.518701 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-kube-api-access-p89tc" pod "acb684d2-d193-4bf7-a07d-4e9c1d4116b2" (UID: "acb684d2-d193-4bf7-a07d-4e9c1d4116b2"). InnerVolumeSpecName "kube-api-access-p89tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:19.617872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.617840 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p89tc\" (UniqueName: \"kubernetes.io/projected/acb684d2-d193-4bf7-a07d-4e9c1d4116b2-kube-api-access-p89tc\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:19.713129 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.713045 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts"] Nov 03 00:15:19.724212 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.724181 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8"] Nov 03 00:15:19.727636 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:19.727607 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-c25940-29894415-pf9x8"] Nov 03 00:15:20.221374 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.221353 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" Nov 03 00:15:20.221766 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.221399 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp" event={"ID":"acb684d2-d193-4bf7-a07d-4e9c1d4116b2","Type":"ContainerDied","Data":"a456eafa06dd772568a815931e76145de7966680e86dc85dd2031eb3148ac3a7"} Nov 03 00:15:20.221766 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.221465 5672 scope.go:122] "RemoveContainer" containerID="0d38ce3c24e14ee52fcec5e21bc56356469114d7beb00270c3aa8e2f5fb30e8c" Nov 03 00:15:20.223923 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.223892 5672 generic.go:356] "Generic (PLEG): container finished" podID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerID="eb8610ac7ff1f4ffe991c7032f72679ce7649dbe7df1f8cd074147e497a3bdd4" exitCode=60 Nov 03 00:15:20.224025 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.223972 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" event={"ID":"7056ae18-f4aa-4cb5-b53b-70bd53e7c477","Type":"ContainerDied","Data":"eb8610ac7ff1f4ffe991c7032f72679ce7649dbe7df1f8cd074147e497a3bdd4"} Nov 03 00:15:20.224385 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.224366 5672 scope.go:122] "RemoveContainer" containerID="eb8610ac7ff1f4ffe991c7032f72679ce7649dbe7df1f8cd074147e497a3bdd4" Nov 03 00:15:20.224648 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:20.224625 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m_odh-ai-gateway-infra(7056ae18-f4aa-4cb5-b53b-70bd53e7c477)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" Nov 03 00:15:20.228126 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.228105 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" Nov 03 00:15:20.228534 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.228111 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g" event={"ID":"3c80c505-2325-46ed-bdb0-95899c8ec5d8","Type":"ContainerDied","Data":"db6d51899fb27c81a182122c734dd292a3dee4191fdb4c7f278fcd78012d48e7"} Nov 03 00:15:20.235035 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.234958 5672 scope.go:122] "RemoveContainer" containerID="7cfc63c34c6aa519770e8b8bc8d1daaa43c0ad4c59547b4734827141d2069c79" Nov 03 00:15:20.255397 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.251652 5672 scope.go:122] "RemoveContainer" containerID="deaf4c2ed986bee35a78d281d15796ad9d4e9b50e30b599c91ea42e86ea6e350" Nov 03 00:15:20.312484 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.312344 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m"] Nov 03 00:15:20.423353 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.423321 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" Nov 03 00:15:20.465912 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.465885 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" Nov 03 00:15:20.469635 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.469577 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" Nov 03 00:15:20.526321 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.526290 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f186952f-5082-4cf7-bad8-a6ae7c4089b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186952f-5082-4cf7-bad8-a6ae7c4089b5-maas-api-service-ca\") pod \"f186952f-5082-4cf7-bad8-a6ae7c4089b5\" (UID: \"f186952f-5082-4cf7-bad8-a6ae7c4089b5\") " Nov 03 00:15:20.526501 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.526468 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f186952f-5082-4cf7-bad8-a6ae7c4089b5-kube-api-access-6wkft\" (UniqueName: \"kubernetes.io/projected/f186952f-5082-4cf7-bad8-a6ae7c4089b5-kube-api-access-6wkft\") pod \"f186952f-5082-4cf7-bad8-a6ae7c4089b5\" (UID: \"f186952f-5082-4cf7-bad8-a6ae7c4089b5\") " Nov 03 00:15:20.526545 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.526511 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c90ca2-20e4-4c85-94de-9e7127186ea9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c90ca2-20e4-4c85-94de-9e7127186ea9-maas-api-service-ca\") pod \"74c90ca2-20e4-4c85-94de-9e7127186ea9\" (UID: \"74c90ca2-20e4-4c85-94de-9e7127186ea9\") " Nov 03 00:15:20.526664 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.526642 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f186952f-5082-4cf7-bad8-a6ae7c4089b5-maas-api-service-ca" pod "f186952f-5082-4cf7-bad8-a6ae7c4089b5" (UID: "f186952f-5082-4cf7-bad8-a6ae7c4089b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:20.526725 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.526646 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59278e2-911a-426e-80dc-7ed972e6fd36-kube-api-access-2bzwg\" (UniqueName: \"kubernetes.io/projected/d59278e2-911a-426e-80dc-7ed972e6fd36-kube-api-access-2bzwg\") pod \"d59278e2-911a-426e-80dc-7ed972e6fd36\" (UID: \"d59278e2-911a-426e-80dc-7ed972e6fd36\") " Nov 03 00:15:20.526777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.526740 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c90ca2-20e4-4c85-94de-9e7127186ea9-kube-api-access-75lmb\" (UniqueName: \"kubernetes.io/projected/74c90ca2-20e4-4c85-94de-9e7127186ea9-kube-api-access-75lmb\") pod \"74c90ca2-20e4-4c85-94de-9e7127186ea9\" (UID: \"74c90ca2-20e4-4c85-94de-9e7127186ea9\") " Nov 03 00:15:20.526890 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.526872 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59278e2-911a-426e-80dc-7ed972e6fd36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59278e2-911a-426e-80dc-7ed972e6fd36-maas-api-service-ca\") pod \"d59278e2-911a-426e-80dc-7ed972e6fd36\" (UID: \"d59278e2-911a-426e-80dc-7ed972e6fd36\") " Nov 03 00:15:20.526989 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.526965 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c90ca2-20e4-4c85-94de-9e7127186ea9-maas-api-service-ca" pod "74c90ca2-20e4-4c85-94de-9e7127186ea9" (UID: "74c90ca2-20e4-4c85-94de-9e7127186ea9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:20.527256 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.527231 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59278e2-911a-426e-80dc-7ed972e6fd36-maas-api-service-ca" pod "d59278e2-911a-426e-80dc-7ed972e6fd36" (UID: "d59278e2-911a-426e-80dc-7ed972e6fd36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:20.527360 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.527308 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59278e2-911a-426e-80dc-7ed972e6fd36-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:20.527360 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.527325 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f186952f-5082-4cf7-bad8-a6ae7c4089b5-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:20.527360 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.527335 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c90ca2-20e4-4c85-94de-9e7127186ea9-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:20.528679 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.528655 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f186952f-5082-4cf7-bad8-a6ae7c4089b5-kube-api-access-6wkft" pod "f186952f-5082-4cf7-bad8-a6ae7c4089b5" (UID: "f186952f-5082-4cf7-bad8-a6ae7c4089b5"). InnerVolumeSpecName "kube-api-access-6wkft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:20.529059 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.529040 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59278e2-911a-426e-80dc-7ed972e6fd36-kube-api-access-2bzwg" pod "d59278e2-911a-426e-80dc-7ed972e6fd36" (UID: "d59278e2-911a-426e-80dc-7ed972e6fd36"). InnerVolumeSpecName "kube-api-access-2bzwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:20.529107 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.529068 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c90ca2-20e4-4c85-94de-9e7127186ea9-kube-api-access-75lmb" pod "74c90ca2-20e4-4c85-94de-9e7127186ea9" (UID: "74c90ca2-20e4-4c85-94de-9e7127186ea9"). InnerVolumeSpecName "kube-api-access-75lmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:20.568318 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.568296 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g"] Nov 03 00:15:20.572017 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.571996 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894415-kph5g"] Nov 03 00:15:20.628727 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.628702 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wkft\" (UniqueName: \"kubernetes.io/projected/f186952f-5082-4cf7-bad8-a6ae7c4089b5-kube-api-access-6wkft\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:20.628727 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.628726 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bzwg\" (UniqueName: \"kubernetes.io/projected/d59278e2-911a-426e-80dc-7ed972e6fd36-kube-api-access-2bzwg\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:20.628879 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.628736 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75lmb\" (UniqueName: \"kubernetes.io/projected/74c90ca2-20e4-4c85-94de-9e7127186ea9-kube-api-access-75lmb\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:20.712281 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.712250 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" path="/var/lib/kubelet/pods/3c80c505-2325-46ed-bdb0-95899c8ec5d8/volumes" Nov 03 00:15:20.712682 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.712669 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583aad32-dafe-46d4-861f-f07566330db8" path="/var/lib/kubelet/pods/583aad32-dafe-46d4-861f-f07566330db8/volumes" Nov 03 00:15:20.713112 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.713090 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" path="/var/lib/kubelet/pods/d82d527a-412c-4e3d-a8fe-afc3557f9ee5/volumes" Nov 03 00:15:20.713927 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.713903 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" podUID="3941edda-d840-46e5-b257-d42202cd8214" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:15:20.885917 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.885883 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp"] Nov 03 00:15:20.889658 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:20.889630 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894415-xkkvp"] Nov 03 00:15:21.245110 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.245076 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" Nov 03 00:15:21.245110 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.245099 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg" event={"ID":"d59278e2-911a-426e-80dc-7ed972e6fd36","Type":"ContainerDied","Data":"9282ee515f9a77556424dc904b506e0d9a9116808af54bda70d09388a76fdba5"} Nov 03 00:15:21.245594 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.245213 5672 scope.go:122] "RemoveContainer" containerID="a419402e084bba5442b707df634d49072068940512e6b1bc0978a798161fa8ec" Nov 03 00:15:21.250891 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.250678 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" event={"ID":"f186952f-5082-4cf7-bad8-a6ae7c4089b5","Type":"ContainerDied","Data":"704a8a46c8fcdd88c0dfbbbc1d7be2ab7a77a950558d94a024a1b4b6bd85d0a2"} Nov 03 00:15:21.250891 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.250704 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc" Nov 03 00:15:21.252282 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.252247 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" event={"ID":"74c90ca2-20e4-4c85-94de-9e7127186ea9","Type":"ContainerDied","Data":"84c0fd8efd6b175d0f32ea417d957ec0e827bea1194b2ac2e7da76362e2f874d"} Nov 03 00:15:21.252411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.252295 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts" Nov 03 00:15:21.260472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.260431 5672 scope.go:122] "RemoveContainer" containerID="a787d6c5faca75f39bd8d1719d47c627eb4ef16c154abc6b8a3391f08605643d" Nov 03 00:15:21.269674 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.269657 5672 scope.go:122] "RemoveContainer" containerID="41bf3b000a24bf743a09bcf75607556738317117f3da6c87efc9e908cba0f8c8" Nov 03 00:15:21.398092 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.398066 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc"] Nov 03 00:15:21.399816 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.399789 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" Nov 03 00:15:21.402031 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.402007 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fb6e1d-29894415-zvblc"] Nov 03 00:15:21.437526 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.437503 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-kube-api-access-qfz4d\" (UniqueName: \"kubernetes.io/projected/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-kube-api-access-qfz4d\") pod \"7056ae18-f4aa-4cb5-b53b-70bd53e7c477\" (UID: \"7056ae18-f4aa-4cb5-b53b-70bd53e7c477\") " Nov 03 00:15:21.437642 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.437578 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-maas-api-service-ca\") pod \"7056ae18-f4aa-4cb5-b53b-70bd53e7c477\" (UID: \"7056ae18-f4aa-4cb5-b53b-70bd53e7c477\") " Nov 03 00:15:21.437934 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.437908 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-maas-api-service-ca" pod "7056ae18-f4aa-4cb5-b53b-70bd53e7c477" (UID: "7056ae18-f4aa-4cb5-b53b-70bd53e7c477"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:21.439591 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.439571 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-kube-api-access-qfz4d" pod "7056ae18-f4aa-4cb5-b53b-70bd53e7c477" (UID: "7056ae18-f4aa-4cb5-b53b-70bd53e7c477"). InnerVolumeSpecName "kube-api-access-qfz4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:21.538619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.538550 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfz4d\" (UniqueName: \"kubernetes.io/projected/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-kube-api-access-qfz4d\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:21.538619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.538576 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7056ae18-f4aa-4cb5-b53b-70bd53e7c477-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:21.609234 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.609202 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg"] Nov 03 00:15:21.610143 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.610121 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-hpvgg"] Nov 03 00:15:21.711967 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.711933 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts"] Nov 03 00:15:21.716039 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.716008 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894415-qk8ts"] Nov 03 00:15:21.995851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:21.995821 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 03 00:15:22.043714 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.043684 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3941edda-d840-46e5-b257-d42202cd8214-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3941edda-d840-46e5-b257-d42202cd8214-plugins-config-volume\") pod \"3941edda-d840-46e5-b257-d42202cd8214\" (UID: \"3941edda-d840-46e5-b257-d42202cd8214\") " Nov 03 00:15:22.043843 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.043832 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3941edda-d840-46e5-b257-d42202cd8214-kube-api-access-djk9c\" (UniqueName: \"kubernetes.io/projected/3941edda-d840-46e5-b257-d42202cd8214-kube-api-access-djk9c\") pod \"3941edda-d840-46e5-b257-d42202cd8214\" (UID: \"3941edda-d840-46e5-b257-d42202cd8214\") " Nov 03 00:15:22.044067 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.044046 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3941edda-d840-46e5-b257-d42202cd8214-plugins-config-volume" pod "3941edda-d840-46e5-b257-d42202cd8214" (UID: "3941edda-d840-46e5-b257-d42202cd8214"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:22.044251 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.044238 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3941edda-d840-46e5-b257-d42202cd8214-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:22.046573 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.046545 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3941edda-d840-46e5-b257-d42202cd8214-kube-api-access-djk9c" pod "3941edda-d840-46e5-b257-d42202cd8214" (UID: "3941edda-d840-46e5-b257-d42202cd8214"). InnerVolumeSpecName "kube-api-access-djk9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:22.145525 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.145436 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djk9c\" (UniqueName: \"kubernetes.io/projected/3941edda-d840-46e5-b257-d42202cd8214-kube-api-access-djk9c\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:22.266280 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.266244 5672 generic.go:356] "Generic (PLEG): container finished" podID="3941edda-d840-46e5-b257-d42202cd8214" containerID="e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee" exitCode=0 Nov 03 00:15:22.266708 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.266288 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" Nov 03 00:15:22.266708 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.266331 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" event={"ID":"3941edda-d840-46e5-b257-d42202cd8214","Type":"ContainerDied","Data":"e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee"} Nov 03 00:15:22.266708 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.266389 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl" event={"ID":"3941edda-d840-46e5-b257-d42202cd8214","Type":"ContainerDied","Data":"7fb5f23139365ac9ab2d4dd0d9a6a5f47ce060ac37e2b3d26a731e3f1dc5bf0f"} Nov 03 00:15:22.266708 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.266420 5672 scope.go:122] "RemoveContainer" containerID="e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee" Nov 03 00:15:22.268236 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.268161 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" Nov 03 00:15:22.268236 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.268195 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m" event={"ID":"7056ae18-f4aa-4cb5-b53b-70bd53e7c477","Type":"ContainerDied","Data":"669b46e80d85467ea964f71829876a379e8be1d1d884ed01427cf345fd83bf6f"} Nov 03 00:15:22.278892 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.278873 5672 scope.go:122] "RemoveContainer" containerID="e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee" Nov 03 00:15:22.279813 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:22.279750 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee\": container with ID starting with e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee not found: ID does not exist" containerID="e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee" Nov 03 00:15:22.279813 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.279790 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee"} err="failed to get container status \"e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee\": rpc error: code = NotFound desc = could not find container \"e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee\": container with ID starting with e6f139f76dd269522fbe27a949a6d7cc26e1f73b8beed06e316666407a197fee not found: ID does not exist" Nov 03 00:15:22.279963 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.279813 5672 scope.go:122] "RemoveContainer" containerID="eb8610ac7ff1f4ffe991c7032f72679ce7649dbe7df1f8cd074147e497a3bdd4" Nov 03 00:15:22.420307 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.420230 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl"] Nov 03 00:15:22.423473 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.423450 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c5148c-6cbcbf955c-cqxhl"] Nov 03 00:15:22.524849 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.524817 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m"] Nov 03 00:15:22.529651 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.529620 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a51378-29894415-6286m"] Nov 03 00:15:22.712798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.712726 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3941edda-d840-46e5-b257-d42202cd8214" path="/var/lib/kubelet/pods/3941edda-d840-46e5-b257-d42202cd8214/volumes" Nov 03 00:15:22.713129 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.713115 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" path="/var/lib/kubelet/pods/7056ae18-f4aa-4cb5-b53b-70bd53e7c477/volumes" Nov 03 00:15:22.713531 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.713518 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" path="/var/lib/kubelet/pods/74c90ca2-20e4-4c85-94de-9e7127186ea9/volumes" Nov 03 00:15:22.714201 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.714184 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" path="/var/lib/kubelet/pods/acb684d2-d193-4bf7-a07d-4e9c1d4116b2/volumes" Nov 03 00:15:22.714597 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.714583 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" path="/var/lib/kubelet/pods/d59278e2-911a-426e-80dc-7ed972e6fd36/volumes" Nov 03 00:15:22.714953 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:22.714936 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" path="/var/lib/kubelet/pods/f186952f-5082-4cf7-bad8-a6ae7c4089b5/volumes" Nov 03 00:15:26.020722 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:26.020683 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:15:29.869942 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:29.869907 5672 scope.go:122] "RemoveContainer" containerID="52c680365d356d3c253beb815ec6b48b34c21329d60574ae3f4736fe218ff2ab" Nov 03 00:15:30.838773 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:30.838735 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:15:35.608901 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:35.608865 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf"] Nov 03 00:15:35.709264 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:35.709231 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7"] Nov 03 00:15:37.987086 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987054 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc"] Nov 03 00:15:37.987540 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987526 5672 cpu_manager.go:414] "containerMap: removing container" podUID="583aad32-dafe-46d4-861f-f07566330db8" containerName="cleanup" Nov 03 00:15:37.987590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987540 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583aad32-dafe-46d4-861f-f07566330db8" containerName="cleanup" Nov 03 00:15:37.987590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987550 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerName="cleanup" Nov 03 00:15:37.987590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987555 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerName="cleanup" Nov 03 00:15:37.987590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987563 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerName="cleanup" Nov 03 00:15:37.987590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987567 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerName="cleanup" Nov 03 00:15:37.987590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987577 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerName="cleanup" Nov 03 00:15:37.987590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987581 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerName="cleanup" Nov 03 00:15:37.987590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987586 5672 cpu_manager.go:414] "containerMap: removing container" podUID="583aad32-dafe-46d4-861f-f07566330db8" containerName="cleanup" Nov 03 00:15:37.987590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987591 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583aad32-dafe-46d4-861f-f07566330db8" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987598 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987603 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987610 5672 cpu_manager.go:414] "containerMap: removing container" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987613 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987621 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987625 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987632 5672 cpu_manager.go:414] "containerMap: removing container" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987636 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987645 5672 cpu_manager.go:414] "containerMap: removing container" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987650 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987656 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987660 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987666 5672 cpu_manager.go:414] "containerMap: removing container" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987671 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987676 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987681 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987687 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987691 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987701 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987705 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987712 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987716 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987725 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987730 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987737 5672 cpu_manager.go:414] "containerMap: removing container" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987742 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987748 5672 cpu_manager.go:414] "containerMap: removing container" podUID="583aad32-dafe-46d4-861f-f07566330db8" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987753 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583aad32-dafe-46d4-861f-f07566330db8" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987770 5672 cpu_manager.go:414] "containerMap: removing container" podUID="2144ab98-3fe5-4849-b80c-507d4e015a46" containerName="revoke-keys" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987774 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2144ab98-3fe5-4849-b80c-507d4e015a46" containerName="revoke-keys" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987793 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3941edda-d840-46e5-b257-d42202cd8214" containerName="payload-processing" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987797 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3941edda-d840-46e5-b257-d42202cd8214" containerName="payload-processing" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987805 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987809 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerName="cleanup" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987879 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3941edda-d840-46e5-b257-d42202cd8214" containerName="payload-processing" Nov 03 00:15:37.987872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987887 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987894 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987900 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987906 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="583aad32-dafe-46d4-861f-f07566330db8" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987911 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987916 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987921 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987926 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="583aad32-dafe-46d4-861f-f07566330db8" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987931 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987935 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987940 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c80c505-2325-46ed-bdb0-95899c8ec5d8" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987945 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987950 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987954 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987960 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987965 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987971 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7056ae18-f4aa-4cb5-b53b-70bd53e7c477" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987976 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerName="cleanup" Nov 03 00:15:37.988990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.987982 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="2144ab98-3fe5-4849-b80c-507d4e015a46" containerName="revoke-keys" Nov 03 00:15:37.991098 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:37.991078 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:38.087018 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.086419 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:15:38.087018 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.086447 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:15:38.087018 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.086457 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:15:38.087018 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.086525 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:15:38.089877 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.089852 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc"] Nov 03 00:15:38.125719 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.125689 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txg5t\" (UniqueName: \"kubernetes.io/projected/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-kube-api-access-txg5t\") pod \"maas-api-revoke-sub-820b1c31-b6rwc\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:38.125902 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.125810 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820b1c31-b6rwc\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:38.125981 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.125939 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820b1c31-b6rwc\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:38.227181 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.227132 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820b1c31-b6rwc\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:38.227354 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.227237 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820b1c31-b6rwc\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:38.227354 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.227349 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txg5t\" (UniqueName: \"kubernetes.io/projected/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-kube-api-access-txg5t\") pod \"maas-api-revoke-sub-820b1c31-b6rwc\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:38.227898 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.227876 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820b1c31-b6rwc\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:38.229582 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.229560 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820b1c31-b6rwc\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:38.281498 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.281443 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txg5t\" (UniqueName: \"kubernetes.io/projected/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-kube-api-access-txg5t\") pod \"maas-api-revoke-sub-820b1c31-b6rwc\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:38.303594 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.303570 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:38.677570 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:38.677538 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc"] Nov 03 00:15:38.680022 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:15:38.679990 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662d88bc_eebb_4c34_8f3a_9f36ea9d1002.slice/crio-4653439af940d460910de9b1d827b88790aacb3a7d67e4a6465ad538517bfa30 WatchSource:0}: Error finding container 4653439af940d460910de9b1d827b88790aacb3a7d67e4a6465ad538517bfa30: Status 404 returned error can't find the container with id 4653439af940d460910de9b1d827b88790aacb3a7d67e4a6465ad538517bfa30 Nov 03 00:15:39.392068 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:39.392030 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" event={"ID":"662d88bc-eebb-4c34-8f3a-9f36ea9d1002","Type":"ContainerStarted","Data":"4f7ddc338d904577a98bc4d61c3f5ebd1c697bafc98db37d91e409ce94369c66"} Nov 03 00:15:39.392508 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:39.392088 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" event={"ID":"662d88bc-eebb-4c34-8f3a-9f36ea9d1002","Type":"ContainerStarted","Data":"4653439af940d460910de9b1d827b88790aacb3a7d67e4a6465ad538517bfa30"} Nov 03 00:15:39.498855 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:39.498799 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" podStartSLOduration=2.4987817039999998 podStartE2EDuration="2.498781704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:39.496518959 +0000 UTC m=+4932.903830789" watchObservedRunningTime="2026-11-03 00:15:39.498781704 +0000 UTC m=+4932.906093540" Nov 03 00:15:40.400580 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:40.400545 5672 generic.go:356] "Generic (PLEG): container finished" podID="662d88bc-eebb-4c34-8f3a-9f36ea9d1002" containerID="4f7ddc338d904577a98bc4d61c3f5ebd1c697bafc98db37d91e409ce94369c66" exitCode=0 Nov 03 00:15:40.400975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:40.400627 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" event={"ID":"662d88bc-eebb-4c34-8f3a-9f36ea9d1002","Type":"ContainerDied","Data":"4f7ddc338d904577a98bc4d61c3f5ebd1c697bafc98db37d91e409ce94369c66"} Nov 03 00:15:40.586461 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:40.586425 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc"] Nov 03 00:15:40.609648 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:40.609608 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" podUID="09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b" containerName="payload-processing" containerID="cri-o://60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba" gracePeriod=25 Nov 03 00:15:40.710540 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:40.710440 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" podUID="11791814-60d7-405a-ae75-ff0b9e96983a" containerName="payload-pre-processing" containerID="cri-o://4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b" gracePeriod=25 Nov 03 00:15:41.571186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:41.571135 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:41.667529 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:41.667429 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-xks-ca\") pod \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " Nov 03 00:15:41.667718 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:41.667546 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-kube-api-access-txg5t\" (UniqueName: \"kubernetes.io/projected/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-kube-api-access-txg5t\") pod \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " Nov 03 00:15:41.667718 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:41.667583 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-openshift-ca\") pod \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\" (UID: \"662d88bc-eebb-4c34-8f3a-9f36ea9d1002\") " Nov 03 00:15:41.667993 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:41.667961 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-openshift-ca" pod "662d88bc-eebb-4c34-8f3a-9f36ea9d1002" (UID: "662d88bc-eebb-4c34-8f3a-9f36ea9d1002"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:41.669611 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:41.669583 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-xks-ca" pod "662d88bc-eebb-4c34-8f3a-9f36ea9d1002" (UID: "662d88bc-eebb-4c34-8f3a-9f36ea9d1002"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:41.669759 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:41.669707 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-kube-api-access-txg5t" pod "662d88bc-eebb-4c34-8f3a-9f36ea9d1002" (UID: "662d88bc-eebb-4c34-8f3a-9f36ea9d1002"). InnerVolumeSpecName "kube-api-access-txg5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:41.769435 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:41.769397 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:41.769435 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:41.769429 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txg5t\" (UniqueName: \"kubernetes.io/projected/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-kube-api-access-txg5t\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:41.769435 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:41.769443 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662d88bc-eebb-4c34-8f3a-9f36ea9d1002-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:42.413804 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:42.413775 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" Nov 03 00:15:42.413990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:42.413813 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc" event={"ID":"662d88bc-eebb-4c34-8f3a-9f36ea9d1002","Type":"ContainerDied","Data":"4653439af940d460910de9b1d827b88790aacb3a7d67e4a6465ad538517bfa30"} Nov 03 00:15:42.413990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:42.413855 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4653439af940d460910de9b1d827b88790aacb3a7d67e4a6465ad538517bfa30" Nov 03 00:15:42.683207 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:42.683108 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" podUID="11791814-60d7-405a-ae75-ff0b9e96983a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:15:42.683207 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:42.683132 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" podUID="09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:15:47.504807 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:47.504762 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx"] Nov 03 00:15:48.807473 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.807440 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh"] Nov 03 00:15:48.807957 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.807947 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerName="cleanup" Nov 03 00:15:48.808047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.807960 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerName="cleanup" Nov 03 00:15:48.808047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.807979 5672 cpu_manager.go:414] "containerMap: removing container" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerName="cleanup" Nov 03 00:15:48.808047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.807984 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerName="cleanup" Nov 03 00:15:48.808047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.807996 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerName="cleanup" Nov 03 00:15:48.808047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808001 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerName="cleanup" Nov 03 00:15:48.808047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808014 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerName="cleanup" Nov 03 00:15:48.808047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808019 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerName="cleanup" Nov 03 00:15:48.808047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808028 5672 cpu_manager.go:414] "containerMap: removing container" podUID="662d88bc-eebb-4c34-8f3a-9f36ea9d1002" containerName="revoke-keys" Nov 03 00:15:48.808047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808032 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662d88bc-eebb-4c34-8f3a-9f36ea9d1002" containerName="revoke-keys" Nov 03 00:15:48.808553 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808102 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82d527a-412c-4e3d-a8fe-afc3557f9ee5" containerName="cleanup" Nov 03 00:15:48.808553 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808111 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="583aad32-dafe-46d4-861f-f07566330db8" containerName="cleanup" Nov 03 00:15:48.808553 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808117 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c90ca2-20e4-4c85-94de-9e7127186ea9" containerName="cleanup" Nov 03 00:15:48.808553 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808124 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59278e2-911a-426e-80dc-7ed972e6fd36" containerName="cleanup" Nov 03 00:15:48.808553 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808131 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f186952f-5082-4cf7-bad8-a6ae7c4089b5" containerName="cleanup" Nov 03 00:15:48.808553 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808137 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="662d88bc-eebb-4c34-8f3a-9f36ea9d1002" containerName="revoke-keys" Nov 03 00:15:48.808553 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.808144 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb684d2-d193-4bf7-a07d-4e9c1d4116b2" containerName="cleanup" Nov 03 00:15:48.811711 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.811690 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:48.864994 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.864965 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh"] Nov 03 00:15:48.907363 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.907310 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:15:48.907363 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.907351 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:15:48.907642 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.907406 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:15:48.907642 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.907486 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:15:48.946387 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.946347 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmj8h\" (UniqueName: \"kubernetes.io/projected/19958d33-9ec4-44ca-98b7-913dc66aecef-kube-api-access-bmj8h\") pod \"maas-api-revoke-sub-b27326ad-6szrh\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:48.946582 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.946424 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b27326ad-6szrh\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:48.946685 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:48.946596 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b27326ad-6szrh\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:49.048087 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:49.048047 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmj8h\" (UniqueName: \"kubernetes.io/projected/19958d33-9ec4-44ca-98b7-913dc66aecef-kube-api-access-bmj8h\") pod \"maas-api-revoke-sub-b27326ad-6szrh\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:49.048290 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:49.048101 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b27326ad-6szrh\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:49.048290 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:49.048198 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b27326ad-6szrh\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:49.048824 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:49.048800 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b27326ad-6szrh\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:49.050614 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:49.050586 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b27326ad-6szrh\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:49.100874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:49.100812 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmj8h\" (UniqueName: \"kubernetes.io/projected/19958d33-9ec4-44ca-98b7-913dc66aecef-kube-api-access-bmj8h\") pod \"maas-api-revoke-sub-b27326ad-6szrh\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:49.128222 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:49.128199 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:49.495222 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:49.495198 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh"] Nov 03 00:15:49.496400 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:15:49.496370 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19958d33_9ec4_44ca_98b7_913dc66aecef.slice/crio-7f0338fbcfcaf113feff7894076b398fe3b657d938a886a7e256429a44eccaf4 WatchSource:0}: Error finding container 7f0338fbcfcaf113feff7894076b398fe3b657d938a886a7e256429a44eccaf4: Status 404 returned error can't find the container with id 7f0338fbcfcaf113feff7894076b398fe3b657d938a886a7e256429a44eccaf4 Nov 03 00:15:50.469820 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:50.469787 5672 generic.go:356] "Generic (PLEG): container finished" podID="19958d33-9ec4-44ca-98b7-913dc66aecef" containerID="99947a9f759fdabcaa95794fd4d150a9eba322c778760503acaea9a27b113197" exitCode=0 Nov 03 00:15:50.470236 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:50.469882 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" event={"ID":"19958d33-9ec4-44ca-98b7-913dc66aecef","Type":"ContainerDied","Data":"99947a9f759fdabcaa95794fd4d150a9eba322c778760503acaea9a27b113197"} Nov 03 00:15:50.470236 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:50.469954 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" event={"ID":"19958d33-9ec4-44ca-98b7-913dc66aecef","Type":"ContainerStarted","Data":"7f0338fbcfcaf113feff7894076b398fe3b657d938a886a7e256429a44eccaf4"} Nov 03 00:15:51.624140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:51.624117 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:51.677946 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:51.677912 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19958d33-9ec4-44ca-98b7-913dc66aecef-kube-api-access-bmj8h\" (UniqueName: \"kubernetes.io/projected/19958d33-9ec4-44ca-98b7-913dc66aecef-kube-api-access-bmj8h\") pod \"19958d33-9ec4-44ca-98b7-913dc66aecef\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " Nov 03 00:15:51.678107 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:51.677979 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-openshift-ca\") pod \"19958d33-9ec4-44ca-98b7-913dc66aecef\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " Nov 03 00:15:51.678107 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:51.678085 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-xks-ca\") pod \"19958d33-9ec4-44ca-98b7-913dc66aecef\" (UID: \"19958d33-9ec4-44ca-98b7-913dc66aecef\") " Nov 03 00:15:51.678450 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:51.678421 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-openshift-ca" pod "19958d33-9ec4-44ca-98b7-913dc66aecef" (UID: "19958d33-9ec4-44ca-98b7-913dc66aecef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:51.678702 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:51.678682 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:51.680003 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:51.679979 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19958d33-9ec4-44ca-98b7-913dc66aecef-kube-api-access-bmj8h" pod "19958d33-9ec4-44ca-98b7-913dc66aecef" (UID: "19958d33-9ec4-44ca-98b7-913dc66aecef"). InnerVolumeSpecName "kube-api-access-bmj8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:51.680135 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:51.680116 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-xks-ca" pod "19958d33-9ec4-44ca-98b7-913dc66aecef" (UID: "19958d33-9ec4-44ca-98b7-913dc66aecef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:51.780092 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:51.780019 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmj8h\" (UniqueName: \"kubernetes.io/projected/19958d33-9ec4-44ca-98b7-913dc66aecef-kube-api-access-bmj8h\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:51.780092 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:51.780045 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19958d33-9ec4-44ca-98b7-913dc66aecef-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:52.484906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:52.484872 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" event={"ID":"19958d33-9ec4-44ca-98b7-913dc66aecef","Type":"ContainerDied","Data":"7f0338fbcfcaf113feff7894076b398fe3b657d938a886a7e256429a44eccaf4"} Nov 03 00:15:52.484906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:52.484900 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh" Nov 03 00:15:52.485118 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:52.484920 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f0338fbcfcaf113feff7894076b398fe3b657d938a886a7e256429a44eccaf4" Nov 03 00:15:52.506234 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:52.506176 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" podUID="5af13ee4-e8a6-4332-9601-4b9f243625b3" containerName="payload-pre-processing" containerID="cri-o://1164a8344a87e5579761d78ed1f3b44cdfb513e7f41bca1481598e7a8be68719" gracePeriod=25 Nov 03 00:15:52.683487 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:52.683452 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" podUID="11791814-60d7-405a-ae75-ff0b9e96983a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:15:52.683487 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:52.683476 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" podUID="09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:15:53.833600 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:53.833564 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" podUID="5af13ee4-e8a6-4332-9601-4b9f243625b3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:15:55.608831 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.608790 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" podUID="8e0cfdca-539c-454d-bead-a56298920ea3" containerName="main" containerID="cri-o://d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e" gracePeriod=45 Nov 03 00:15:55.884556 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.884526 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:15:55.921689 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.921666 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-tmp-dir\") pod \"8e0cfdca-539c-454d-bead-a56298920ea3\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " Nov 03 00:15:55.921817 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.921754 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0cfdca-539c-454d-bead-a56298920ea3-kube-api-access-5zgsm\" (UniqueName: \"kubernetes.io/projected/8e0cfdca-539c-454d-bead-a56298920ea3-kube-api-access-5zgsm\") pod \"8e0cfdca-539c-454d-bead-a56298920ea3\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " Nov 03 00:15:55.921817 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.921774 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-home\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-home\") pod \"8e0cfdca-539c-454d-bead-a56298920ea3\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " Nov 03 00:15:55.921817 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.921798 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-dshm\") pod \"8e0cfdca-539c-454d-bead-a56298920ea3\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " Nov 03 00:15:55.921991 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.921833 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e0cfdca-539c-454d-bead-a56298920ea3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0cfdca-539c-454d-bead-a56298920ea3-tls-certs\") pod \"8e0cfdca-539c-454d-bead-a56298920ea3\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " Nov 03 00:15:55.921991 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.921905 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-model-cache\") pod \"8e0cfdca-539c-454d-bead-a56298920ea3\" (UID: \"8e0cfdca-539c-454d-bead-a56298920ea3\") " Nov 03 00:15:55.921991 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.921911 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-tmp-dir" pod "8e0cfdca-539c-454d-bead-a56298920ea3" (UID: "8e0cfdca-539c-454d-bead-a56298920ea3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:55.922267 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.922041 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-home" pod "8e0cfdca-539c-454d-bead-a56298920ea3" (UID: "8e0cfdca-539c-454d-bead-a56298920ea3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:55.922267 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.922190 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-model-cache" pod "8e0cfdca-539c-454d-bead-a56298920ea3" (UID: "8e0cfdca-539c-454d-bead-a56298920ea3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:55.922431 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.922412 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:55.922481 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.922439 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:55.922481 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.922454 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:55.923944 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.923916 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0cfdca-539c-454d-bead-a56298920ea3-tls-certs" pod "8e0cfdca-539c-454d-bead-a56298920ea3" (UID: "8e0cfdca-539c-454d-bead-a56298920ea3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:55.924064 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.923924 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-dshm" pod "8e0cfdca-539c-454d-bead-a56298920ea3" (UID: "8e0cfdca-539c-454d-bead-a56298920ea3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:55.924064 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:55.923952 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0cfdca-539c-454d-bead-a56298920ea3-kube-api-access-5zgsm" pod "8e0cfdca-539c-454d-bead-a56298920ea3" (UID: "8e0cfdca-539c-454d-bead-a56298920ea3"). InnerVolumeSpecName "kube-api-access-5zgsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:56.023374 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:56.023333 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:15:56.023608 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.023469 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0cfdca-539c-454d-bead-a56298920ea3-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:56.023739 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.023625 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zgsm\" (UniqueName: \"kubernetes.io/projected/8e0cfdca-539c-454d-bead-a56298920ea3-kube-api-access-5zgsm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:56.023739 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.023644 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0cfdca-539c-454d-bead-a56298920ea3-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:15:56.511401 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.511367 5672 generic.go:356] "Generic (PLEG): container finished" podID="8e0cfdca-539c-454d-bead-a56298920ea3" containerID="d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e" exitCode=0 Nov 03 00:15:56.511592 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.511419 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" Nov 03 00:15:56.511592 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.511459 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" event={"ID":"8e0cfdca-539c-454d-bead-a56298920ea3","Type":"ContainerDied","Data":"d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e"} Nov 03 00:15:56.511592 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.511508 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc" event={"ID":"8e0cfdca-539c-454d-bead-a56298920ea3","Type":"ContainerDied","Data":"21bd7a0584db2b3830bc0248e7f8a86b782657f3e4311d786f33281c6dfbccf0"} Nov 03 00:15:56.511592 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.511525 5672 scope.go:122] "RemoveContainer" containerID="d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e" Nov 03 00:15:56.522214 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.522199 5672 scope.go:122] "RemoveContainer" containerID="d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e" Nov 03 00:15:56.522441 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:15:56.522425 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e\": container with ID starting with d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e not found: ID does not exist" containerID="d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e" Nov 03 00:15:56.522497 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.522449 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e"} err="failed to get container status \"d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e\": rpc error: code = NotFound desc = could not find container \"d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e\": container with ID starting with d2f91a72bc3582f6f06550f62d82f807ab25f06f669f5d5d54ff0c35eabc610e not found: ID does not exist" Nov 03 00:15:56.663703 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.663668 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc"] Nov 03 00:15:56.713477 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:56.713449 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-c25940/rate-test-model-c25940-kserve-6d646cfcc-m55sc"] Nov 03 00:15:58.715559 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:58.715496 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0cfdca-539c-454d-bead-a56298920ea3" path="/var/lib/kubelet/pods/8e0cfdca-539c-454d-bead-a56298920ea3/volumes" Nov 03 00:15:58.887070 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:15:58.887036 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:16:00.371221 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.371188 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd"] Nov 03 00:16:00.371696 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.371683 5672 cpu_manager.go:414] "containerMap: removing container" podUID="19958d33-9ec4-44ca-98b7-913dc66aecef" containerName="revoke-keys" Nov 03 00:16:00.371748 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.371696 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19958d33-9ec4-44ca-98b7-913dc66aecef" containerName="revoke-keys" Nov 03 00:16:00.371748 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.371709 5672 cpu_manager.go:414] "containerMap: removing container" podUID="8e0cfdca-539c-454d-bead-a56298920ea3" containerName="main" Nov 03 00:16:00.371748 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.371715 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0cfdca-539c-454d-bead-a56298920ea3" containerName="main" Nov 03 00:16:00.371853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.371793 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="19958d33-9ec4-44ca-98b7-913dc66aecef" containerName="revoke-keys" Nov 03 00:16:00.371853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.371803 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0cfdca-539c-454d-bead-a56298920ea3" containerName="main" Nov 03 00:16:00.379050 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.379023 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:00.435428 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.435390 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd"] Nov 03 00:16:00.470013 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.469966 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:16:00.470013 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.469991 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:16:00.470285 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.470044 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:16:00.470285 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.470067 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:16:00.484877 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.484851 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a62ed18-nzdqd\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:00.485013 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.484992 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a62ed18-nzdqd\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:00.485126 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.485075 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57bjd\" (UniqueName: \"kubernetes.io/projected/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-kube-api-access-57bjd\") pod \"maas-api-revoke-sub-0a62ed18-nzdqd\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:00.585793 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.585764 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a62ed18-nzdqd\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:00.585975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.585805 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57bjd\" (UniqueName: \"kubernetes.io/projected/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-kube-api-access-57bjd\") pod \"maas-api-revoke-sub-0a62ed18-nzdqd\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:00.585975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.585869 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a62ed18-nzdqd\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:00.586615 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.586593 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a62ed18-nzdqd\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:00.588385 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.588361 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a62ed18-nzdqd\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:00.638176 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.638135 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57bjd\" (UniqueName: \"kubernetes.io/projected/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-kube-api-access-57bjd\") pod \"maas-api-revoke-sub-0a62ed18-nzdqd\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:00.692898 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.692872 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:00.902612 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.902588 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 03 00:16:00.912677 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.912656 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 03 00:16:00.990030 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.990001 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11791814-60d7-405a-ae75-ff0b9e96983a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11791814-60d7-405a-ae75-ff0b9e96983a-plugins-config-volume\") pod \"11791814-60d7-405a-ae75-ff0b9e96983a\" (UID: \"11791814-60d7-405a-ae75-ff0b9e96983a\") " Nov 03 00:16:00.990199 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.990083 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-kube-api-access-4dtsf\" (UniqueName: \"kubernetes.io/projected/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-kube-api-access-4dtsf\") pod \"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b\" (UID: \"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b\") " Nov 03 00:16:00.990199 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.990140 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11791814-60d7-405a-ae75-ff0b9e96983a-kube-api-access-vr9vt\" (UniqueName: \"kubernetes.io/projected/11791814-60d7-405a-ae75-ff0b9e96983a-kube-api-access-vr9vt\") pod \"11791814-60d7-405a-ae75-ff0b9e96983a\" (UID: \"11791814-60d7-405a-ae75-ff0b9e96983a\") " Nov 03 00:16:00.990284 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.990220 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-plugins-config-volume\") pod \"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b\" (UID: \"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b\") " Nov 03 00:16:00.990442 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.990415 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11791814-60d7-405a-ae75-ff0b9e96983a-plugins-config-volume" pod "11791814-60d7-405a-ae75-ff0b9e96983a" (UID: "11791814-60d7-405a-ae75-ff0b9e96983a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:00.990707 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.990659 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-plugins-config-volume" pod "09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b" (UID: "09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:00.990707 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.990672 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11791814-60d7-405a-ae75-ff0b9e96983a-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:00.992400 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.992371 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-kube-api-access-4dtsf" pod "09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b" (UID: "09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b"). InnerVolumeSpecName "kube-api-access-4dtsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:00.992495 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:00.992408 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11791814-60d7-405a-ae75-ff0b9e96983a-kube-api-access-vr9vt" pod "11791814-60d7-405a-ae75-ff0b9e96983a" (UID: "11791814-60d7-405a-ae75-ff0b9e96983a"). InnerVolumeSpecName "kube-api-access-vr9vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:01.082172 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.082118 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd"] Nov 03 00:16:01.083691 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:16:01.083660 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d39c8a_29cd_4a9e_8e70_5e7c299b7892.slice/crio-7d347b10abb25e19de65bcb541d19571f16fb04786295a0f7b32fd8f2855d6a9 WatchSource:0}: Error finding container 7d347b10abb25e19de65bcb541d19571f16fb04786295a0f7b32fd8f2855d6a9: Status 404 returned error can't find the container with id 7d347b10abb25e19de65bcb541d19571f16fb04786295a0f7b32fd8f2855d6a9 Nov 03 00:16:01.091762 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.091735 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr9vt\" (UniqueName: \"kubernetes.io/projected/11791814-60d7-405a-ae75-ff0b9e96983a-kube-api-access-vr9vt\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:01.091845 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.091764 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:01.091845 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.091776 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dtsf\" (UniqueName: \"kubernetes.io/projected/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b-kube-api-access-4dtsf\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:01.554271 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.554192 5672 generic.go:356] "Generic (PLEG): container finished" podID="09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b" containerID="60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba" exitCode=0 Nov 03 00:16:01.554271 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.554237 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" Nov 03 00:16:01.554764 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.554270 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" event={"ID":"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b","Type":"ContainerDied","Data":"60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba"} Nov 03 00:16:01.554764 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.554346 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf" event={"ID":"09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b","Type":"ContainerDied","Data":"2b7798b09f599a61dbd098e2d0266459b78011b6dbfd63d2d2945b88e8b71282"} Nov 03 00:16:01.554764 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.554403 5672 scope.go:122] "RemoveContainer" containerID="60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba" Nov 03 00:16:01.559256 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.556828 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" event={"ID":"09d39c8a-29cd-4a9e-8e70-5e7c299b7892","Type":"ContainerStarted","Data":"8d0566ac2a4e1c065f81c3436fd13d06852c385ff5288d552398267079e8a973"} Nov 03 00:16:01.559256 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.556932 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" event={"ID":"09d39c8a-29cd-4a9e-8e70-5e7c299b7892","Type":"ContainerStarted","Data":"7d347b10abb25e19de65bcb541d19571f16fb04786295a0f7b32fd8f2855d6a9"} Nov 03 00:16:01.562996 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.562969 5672 generic.go:356] "Generic (PLEG): container finished" podID="11791814-60d7-405a-ae75-ff0b9e96983a" containerID="4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b" exitCode=0 Nov 03 00:16:01.563125 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.563055 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" event={"ID":"11791814-60d7-405a-ae75-ff0b9e96983a","Type":"ContainerDied","Data":"4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b"} Nov 03 00:16:01.563125 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.563100 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" event={"ID":"11791814-60d7-405a-ae75-ff0b9e96983a","Type":"ContainerDied","Data":"5a769d4ac7ef0da8daea7d35d98cdecff9c3ecef3ef94c076fc3b5109282890e"} Nov 03 00:16:01.563297 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.563211 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7" Nov 03 00:16:01.568375 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.568301 5672 scope.go:122] "RemoveContainer" containerID="60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba" Nov 03 00:16:01.568695 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:16:01.568667 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba\": container with ID starting with 60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba not found: ID does not exist" containerID="60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba" Nov 03 00:16:01.568780 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.568711 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba"} err="failed to get container status \"60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba\": rpc error: code = NotFound desc = could not find container \"60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba\": container with ID starting with 60b1211d3c69d89e48e8509daa750641d01e898107fde84a12541088a0af6aba not found: ID does not exist" Nov 03 00:16:01.568780 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.568740 5672 scope.go:122] "RemoveContainer" containerID="4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b" Nov 03 00:16:01.579411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.579390 5672 scope.go:122] "RemoveContainer" containerID="4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b" Nov 03 00:16:01.579703 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:16:01.579679 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b\": container with ID starting with 4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b not found: ID does not exist" containerID="4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b" Nov 03 00:16:01.579775 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.579713 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b"} err="failed to get container status \"4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b\": rpc error: code = NotFound desc = could not find container \"4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b\": container with ID starting with 4687551c0fd3aa859de0e57a7122ee98dc2874e11d7ee61466b2143d4376077b not found: ID does not exist" Nov 03 00:16:01.709275 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.709227 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" podStartSLOduration=1.70921316 podStartE2EDuration="1.70921316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:01.707999084 +0000 UTC m=+4955.115310912" watchObservedRunningTime="2026-11-03 00:16:01.70921316 +0000 UTC m=+4955.116524988" Nov 03 00:16:01.812188 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.812082 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf"] Nov 03 00:16:01.815007 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.814976 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fbec82-555976f77f-stmwf"] Nov 03 00:16:01.916299 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.916270 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7"] Nov 03 00:16:01.923039 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:01.923013 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fbec82-74d7964dcf-s5sn7"] Nov 03 00:16:02.571957 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:02.571923 5672 generic.go:356] "Generic (PLEG): container finished" podID="09d39c8a-29cd-4a9e-8e70-5e7c299b7892" containerID="8d0566ac2a4e1c065f81c3436fd13d06852c385ff5288d552398267079e8a973" exitCode=0 Nov 03 00:16:02.572409 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:02.571965 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" event={"ID":"09d39c8a-29cd-4a9e-8e70-5e7c299b7892","Type":"ContainerDied","Data":"8d0566ac2a4e1c065f81c3436fd13d06852c385ff5288d552398267079e8a973"} Nov 03 00:16:02.712669 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:02.712636 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b" path="/var/lib/kubelet/pods/09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b/volumes" Nov 03 00:16:02.713076 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:02.713054 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11791814-60d7-405a-ae75-ff0b9e96983a" path="/var/lib/kubelet/pods/11791814-60d7-405a-ae75-ff0b9e96983a/volumes" Nov 03 00:16:03.733506 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:03.733482 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:03.821706 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:03.821674 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-kube-api-access-57bjd\" (UniqueName: \"kubernetes.io/projected/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-kube-api-access-57bjd\") pod \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " Nov 03 00:16:03.821863 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:03.821718 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-xks-ca\") pod \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " Nov 03 00:16:03.821863 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:03.821755 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-openshift-ca\") pod \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\" (UID: \"09d39c8a-29cd-4a9e-8e70-5e7c299b7892\") " Nov 03 00:16:03.822201 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:03.822172 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-openshift-ca" pod "09d39c8a-29cd-4a9e-8e70-5e7c299b7892" (UID: "09d39c8a-29cd-4a9e-8e70-5e7c299b7892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:03.823815 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:03.823792 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-xks-ca" pod "09d39c8a-29cd-4a9e-8e70-5e7c299b7892" (UID: "09d39c8a-29cd-4a9e-8e70-5e7c299b7892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:03.823970 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:03.823948 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-kube-api-access-57bjd" pod "09d39c8a-29cd-4a9e-8e70-5e7c299b7892" (UID: "09d39c8a-29cd-4a9e-8e70-5e7c299b7892"). InnerVolumeSpecName "kube-api-access-57bjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:03.833580 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:03.833552 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" podUID="5af13ee4-e8a6-4332-9601-4b9f243625b3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:16:03.923558 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:03.923525 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57bjd\" (UniqueName: \"kubernetes.io/projected/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-kube-api-access-57bjd\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:03.923558 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:03.923553 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:03.923558 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:03.923562 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d39c8a-29cd-4a9e-8e70-5e7c299b7892-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:04.586549 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:04.586520 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" Nov 03 00:16:04.586750 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:04.586515 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd" event={"ID":"09d39c8a-29cd-4a9e-8e70-5e7c299b7892","Type":"ContainerDied","Data":"7d347b10abb25e19de65bcb541d19571f16fb04786295a0f7b32fd8f2855d6a9"} Nov 03 00:16:04.586750 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:04.586649 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d347b10abb25e19de65bcb541d19571f16fb04786295a0f7b32fd8f2855d6a9" Nov 03 00:16:11.241450 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.241360 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt"] Nov 03 00:16:11.242058 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.242040 5672 cpu_manager.go:414] "containerMap: removing container" podUID="09d39c8a-29cd-4a9e-8e70-5e7c299b7892" containerName="revoke-keys" Nov 03 00:16:11.242126 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.242059 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d39c8a-29cd-4a9e-8e70-5e7c299b7892" containerName="revoke-keys" Nov 03 00:16:11.242126 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.242078 5672 cpu_manager.go:414] "containerMap: removing container" podUID="11791814-60d7-405a-ae75-ff0b9e96983a" containerName="payload-pre-processing" Nov 03 00:16:11.242126 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.242086 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11791814-60d7-405a-ae75-ff0b9e96983a" containerName="payload-pre-processing" Nov 03 00:16:11.242126 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.242117 5672 cpu_manager.go:414] "containerMap: removing container" podUID="09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b" containerName="payload-processing" Nov 03 00:16:11.242126 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.242125 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b" containerName="payload-processing" Nov 03 00:16:11.242457 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.242274 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b9f6bc-bcbb-4eec-b5c0-34aadfe1f10b" containerName="payload-processing" Nov 03 00:16:11.242457 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.242295 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d39c8a-29cd-4a9e-8e70-5e7c299b7892" containerName="revoke-keys" Nov 03 00:16:11.242457 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.242305 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="11791814-60d7-405a-ae75-ff0b9e96983a" containerName="payload-pre-processing" Nov 03 00:16:11.246100 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.246078 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:11.295874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.295840 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt"] Nov 03 00:16:11.339162 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.339117 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:16:11.339336 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.339205 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:16:11.368843 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.368815 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:16:11.385662 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.385641 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:16:11.400186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.400133 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f78cd7-lb7tt\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:11.400317 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.400302 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f78cd7-lb7tt\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:11.400368 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.400345 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84t7c\" (UniqueName: \"kubernetes.io/projected/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-kube-api-access-84t7c\") pod \"maas-api-revoke-sub-47f78cd7-lb7tt\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:11.501198 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.501119 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f78cd7-lb7tt\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:11.501198 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.501198 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84t7c\" (UniqueName: \"kubernetes.io/projected/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-kube-api-access-84t7c\") pod \"maas-api-revoke-sub-47f78cd7-lb7tt\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:11.501384 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.501263 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f78cd7-lb7tt\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:11.501916 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.501891 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f78cd7-lb7tt\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:11.503450 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.503430 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f78cd7-lb7tt\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:11.553819 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.553791 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84t7c\" (UniqueName: \"kubernetes.io/projected/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-kube-api-access-84t7c\") pod \"maas-api-revoke-sub-47f78cd7-lb7tt\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:11.559639 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.559621 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:11.923341 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:11.923317 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt"] Nov 03 00:16:11.925316 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:16:11.925280 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7646e0_c37d_4cef_bc05_55e3c2e9f891.slice/crio-9a3dd5c3c76f28073cce91ccc87d71def96538dc0594b6ab6f91f97e71101b1e WatchSource:0}: Error finding container 9a3dd5c3c76f28073cce91ccc87d71def96538dc0594b6ab6f91f97e71101b1e: Status 404 returned error can't find the container with id 9a3dd5c3c76f28073cce91ccc87d71def96538dc0594b6ab6f91f97e71101b1e Nov 03 00:16:12.644511 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:12.644469 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" event={"ID":"0b7646e0-c37d-4cef-bc05-55e3c2e9f891","Type":"ContainerStarted","Data":"c2745dbe2b7060769a077d5eaaaeb1daa7cc3f0744a2862d0740cce7be3ac021"} Nov 03 00:16:12.644931 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:12.644537 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" event={"ID":"0b7646e0-c37d-4cef-bc05-55e3c2e9f891","Type":"ContainerStarted","Data":"9a3dd5c3c76f28073cce91ccc87d71def96538dc0594b6ab6f91f97e71101b1e"} Nov 03 00:16:12.646609 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:12.646584 5672 generic.go:356] "Generic (PLEG): container finished" podID="5af13ee4-e8a6-4332-9601-4b9f243625b3" containerID="1164a8344a87e5579761d78ed1f3b44cdfb513e7f41bca1481598e7a8be68719" exitCode=0 Nov 03 00:16:12.646733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:12.646645 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" event={"ID":"5af13ee4-e8a6-4332-9601-4b9f243625b3","Type":"ContainerDied","Data":"1164a8344a87e5579761d78ed1f3b44cdfb513e7f41bca1481598e7a8be68719"} Nov 03 00:16:12.751865 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:12.751808 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" podStartSLOduration=1.751791314 podStartE2EDuration="1.751791314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:12.748967459 +0000 UTC m=+4966.156279287" watchObservedRunningTime="2026-11-03 00:16:12.751791314 +0000 UTC m=+4966.159103144" Nov 03 00:16:12.766582 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:12.766523 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 03 00:16:12.917096 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:12.917067 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af13ee4-e8a6-4332-9601-4b9f243625b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af13ee4-e8a6-4332-9601-4b9f243625b3-plugins-config-volume\") pod \"5af13ee4-e8a6-4332-9601-4b9f243625b3\" (UID: \"5af13ee4-e8a6-4332-9601-4b9f243625b3\") " Nov 03 00:16:12.917293 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:12.917221 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af13ee4-e8a6-4332-9601-4b9f243625b3-kube-api-access-tz8k9\" (UniqueName: \"kubernetes.io/projected/5af13ee4-e8a6-4332-9601-4b9f243625b3-kube-api-access-tz8k9\") pod \"5af13ee4-e8a6-4332-9601-4b9f243625b3\" (UID: \"5af13ee4-e8a6-4332-9601-4b9f243625b3\") " Nov 03 00:16:12.917533 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:12.917507 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af13ee4-e8a6-4332-9601-4b9f243625b3-plugins-config-volume" pod "5af13ee4-e8a6-4332-9601-4b9f243625b3" (UID: "5af13ee4-e8a6-4332-9601-4b9f243625b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:12.917733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:12.917717 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af13ee4-e8a6-4332-9601-4b9f243625b3-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:12.919301 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:12.919280 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af13ee4-e8a6-4332-9601-4b9f243625b3-kube-api-access-tz8k9" pod "5af13ee4-e8a6-4332-9601-4b9f243625b3" (UID: "5af13ee4-e8a6-4332-9601-4b9f243625b3"). InnerVolumeSpecName "kube-api-access-tz8k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:13.018924 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:13.018855 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz8k9\" (UniqueName: \"kubernetes.io/projected/5af13ee4-e8a6-4332-9601-4b9f243625b3-kube-api-access-tz8k9\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:13.657952 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:13.657916 5672 generic.go:356] "Generic (PLEG): container finished" podID="0b7646e0-c37d-4cef-bc05-55e3c2e9f891" containerID="c2745dbe2b7060769a077d5eaaaeb1daa7cc3f0744a2862d0740cce7be3ac021" exitCode=0 Nov 03 00:16:13.658543 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:13.658001 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" event={"ID":"0b7646e0-c37d-4cef-bc05-55e3c2e9f891","Type":"ContainerDied","Data":"c2745dbe2b7060769a077d5eaaaeb1daa7cc3f0744a2862d0740cce7be3ac021"} Nov 03 00:16:13.659717 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:13.659687 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" event={"ID":"5af13ee4-e8a6-4332-9601-4b9f243625b3","Type":"ContainerDied","Data":"5092b63635d621951353e9cce19de18403ae269045ca176085168a1d5b9df0d2"} Nov 03 00:16:13.659866 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:13.659732 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx" Nov 03 00:16:13.659866 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:13.659748 5672 scope.go:122] "RemoveContainer" containerID="1164a8344a87e5579761d78ed1f3b44cdfb513e7f41bca1481598e7a8be68719" Nov 03 00:16:13.870019 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:13.869883 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx"] Nov 03 00:16:13.873854 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:13.873826 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe63ac-857b6f94bc-qlgqx"] Nov 03 00:16:14.712836 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:14.712798 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af13ee4-e8a6-4332-9601-4b9f243625b3" path="/var/lib/kubelet/pods/5af13ee4-e8a6-4332-9601-4b9f243625b3/volumes" Nov 03 00:16:14.816862 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:14.816840 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:14.938971 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:14.938884 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-xks-ca\") pod \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " Nov 03 00:16:14.938971 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:14.938946 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-kube-api-access-84t7c\" (UniqueName: \"kubernetes.io/projected/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-kube-api-access-84t7c\") pod \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " Nov 03 00:16:14.939239 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:14.939100 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-openshift-ca\") pod \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\" (UID: \"0b7646e0-c37d-4cef-bc05-55e3c2e9f891\") " Nov 03 00:16:14.939719 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:14.939687 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-openshift-ca" pod "0b7646e0-c37d-4cef-bc05-55e3c2e9f891" (UID: "0b7646e0-c37d-4cef-bc05-55e3c2e9f891"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:14.941596 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:14.941559 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-xks-ca" pod "0b7646e0-c37d-4cef-bc05-55e3c2e9f891" (UID: "0b7646e0-c37d-4cef-bc05-55e3c2e9f891"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:14.945591 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:14.945561 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-kube-api-access-84t7c" pod "0b7646e0-c37d-4cef-bc05-55e3c2e9f891" (UID: "0b7646e0-c37d-4cef-bc05-55e3c2e9f891"). InnerVolumeSpecName "kube-api-access-84t7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:15.049206 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:15.041694 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:15.049206 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:15.041743 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:15.049206 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:15.041795 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84t7c\" (UniqueName: \"kubernetes.io/projected/0b7646e0-c37d-4cef-bc05-55e3c2e9f891-kube-api-access-84t7c\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:15.676961 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:15.676928 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" Nov 03 00:16:15.677165 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:15.676958 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt" event={"ID":"0b7646e0-c37d-4cef-bc05-55e3c2e9f891","Type":"ContainerDied","Data":"9a3dd5c3c76f28073cce91ccc87d71def96538dc0594b6ab6f91f97e71101b1e"} Nov 03 00:16:15.677165 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:15.677034 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a3dd5c3c76f28073cce91ccc87d71def96538dc0594b6ab6f91f97e71101b1e" Nov 03 00:16:26.024088 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:16:26.024047 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:16:32.880216 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:32.880182 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh"] Nov 03 00:16:32.880813 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:32.880474 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" containerID="cri-o://2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110" gracePeriod=30 Nov 03 00:16:33.908780 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:33.908742 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h"] Nov 03 00:16:33.909349 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:33.909322 5672 cpu_manager.go:414] "containerMap: removing container" podUID="5af13ee4-e8a6-4332-9601-4b9f243625b3" containerName="payload-pre-processing" Nov 03 00:16:33.909349 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:33.909348 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af13ee4-e8a6-4332-9601-4b9f243625b3" containerName="payload-pre-processing" Nov 03 00:16:33.909573 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:33.909433 5672 cpu_manager.go:414] "containerMap: removing container" podUID="0b7646e0-c37d-4cef-bc05-55e3c2e9f891" containerName="revoke-keys" Nov 03 00:16:33.909573 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:33.909442 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7646e0-c37d-4cef-bc05-55e3c2e9f891" containerName="revoke-keys" Nov 03 00:16:33.909573 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:33.909545 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7646e0-c37d-4cef-bc05-55e3c2e9f891" containerName="revoke-keys" Nov 03 00:16:33.909573 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:33.909557 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af13ee4-e8a6-4332-9601-4b9f243625b3" containerName="payload-pre-processing" Nov 03 00:16:33.914395 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:33.914375 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:33.965257 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:33.965212 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h"] Nov 03 00:16:34.006753 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.006714 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:16:34.006925 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.006910 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:16:34.006977 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.006947 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:16:34.009075 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.009058 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:16:34.068965 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.068931 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de6af444-qmj4h\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:34.068965 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.068972 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zlt2\" (UniqueName: \"kubernetes.io/projected/424d3364-432e-4dde-ac07-75108c4ca592-kube-api-access-6zlt2\") pod \"maas-api-revoke-sub-de6af444-qmj4h\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:34.069241 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.069015 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de6af444-qmj4h\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:34.170831 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.170729 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de6af444-qmj4h\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:34.170831 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.170786 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zlt2\" (UniqueName: \"kubernetes.io/projected/424d3364-432e-4dde-ac07-75108c4ca592-kube-api-access-6zlt2\") pod \"maas-api-revoke-sub-de6af444-qmj4h\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:34.171088 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.170839 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de6af444-qmj4h\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:34.171517 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.171490 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de6af444-qmj4h\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:34.173732 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.173711 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de6af444-qmj4h\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:34.224378 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.224345 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zlt2\" (UniqueName: \"kubernetes.io/projected/424d3364-432e-4dde-ac07-75108c4ca592-kube-api-access-6zlt2\") pod \"maas-api-revoke-sub-de6af444-qmj4h\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:34.228113 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.228095 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:34.600676 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.600644 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h"] Nov 03 00:16:34.602025 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:16:34.601998 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424d3364_432e_4dde_ac07_75108c4ca592.slice/crio-536d7d3993f1869a25294852a77b6f04f2d4e0d66d68ed448d357fb19f8d3290 WatchSource:0}: Error finding container 536d7d3993f1869a25294852a77b6f04f2d4e0d66d68ed448d357fb19f8d3290: Status 404 returned error can't find the container with id 536d7d3993f1869a25294852a77b6f04f2d4e0d66d68ed448d357fb19f8d3290 Nov 03 00:16:34.798118 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:34.798076 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" event={"ID":"424d3364-432e-4dde-ac07-75108c4ca592","Type":"ContainerStarted","Data":"536d7d3993f1869a25294852a77b6f04f2d4e0d66d68ed448d357fb19f8d3290"} Nov 03 00:16:35.027912 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:35.027876 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:16:35.655997 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:35.655957 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw"] Nov 03 00:16:35.707549 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:35.707520 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj"] Nov 03 00:16:35.807649 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:35.807523 5672 generic.go:356] "Generic (PLEG): container finished" podID="424d3364-432e-4dde-ac07-75108c4ca592" containerID="281215eb20e0e81426abf4fc96b0ef7d551aee3d4663397a66cf5fc0e7a124cd" exitCode=0 Nov 03 00:16:35.807848 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:35.807659 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" event={"ID":"424d3364-432e-4dde-ac07-75108c4ca592","Type":"ContainerDied","Data":"281215eb20e0e81426abf4fc96b0ef7d551aee3d4663397a66cf5fc0e7a124cd"} Nov 03 00:16:36.965379 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:36.965343 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:37.120886 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.120786 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-xks-ca\") pod \"424d3364-432e-4dde-ac07-75108c4ca592\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " Nov 03 00:16:37.121049 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.120993 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424d3364-432e-4dde-ac07-75108c4ca592-kube-api-access-6zlt2\" (UniqueName: \"kubernetes.io/projected/424d3364-432e-4dde-ac07-75108c4ca592-kube-api-access-6zlt2\") pod \"424d3364-432e-4dde-ac07-75108c4ca592\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " Nov 03 00:16:37.121135 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.121113 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-openshift-ca\") pod \"424d3364-432e-4dde-ac07-75108c4ca592\" (UID: \"424d3364-432e-4dde-ac07-75108c4ca592\") " Nov 03 00:16:37.121504 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.121476 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-openshift-ca" pod "424d3364-432e-4dde-ac07-75108c4ca592" (UID: "424d3364-432e-4dde-ac07-75108c4ca592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:37.121643 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.121626 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:37.123025 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.122999 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-xks-ca" pod "424d3364-432e-4dde-ac07-75108c4ca592" (UID: "424d3364-432e-4dde-ac07-75108c4ca592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:37.123422 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.123404 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424d3364-432e-4dde-ac07-75108c4ca592-kube-api-access-6zlt2" pod "424d3364-432e-4dde-ac07-75108c4ca592" (UID: "424d3364-432e-4dde-ac07-75108c4ca592"). InnerVolumeSpecName "kube-api-access-6zlt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:37.222493 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.222451 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zlt2\" (UniqueName: \"kubernetes.io/projected/424d3364-432e-4dde-ac07-75108c4ca592-kube-api-access-6zlt2\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:37.222672 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.222511 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424d3364-432e-4dde-ac07-75108c4ca592-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:37.821752 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.821719 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" Nov 03 00:16:37.821963 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.821753 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h" event={"ID":"424d3364-432e-4dde-ac07-75108c4ca592","Type":"ContainerDied","Data":"536d7d3993f1869a25294852a77b6f04f2d4e0d66d68ed448d357fb19f8d3290"} Nov 03 00:16:37.821963 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:37.821799 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="536d7d3993f1869a25294852a77b6f04f2d4e0d66d68ed448d357fb19f8d3290" Nov 03 00:16:38.173808 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.173774 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 03 00:16:38.335051 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.335010 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb"] Nov 03 00:16:38.335424 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.335390 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-podinfo\") pod \"7ed3e348-d810-43a0-afa2-cf8083bc8874\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " Nov 03 00:16:38.335589 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.335448 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-data\") pod \"7ed3e348-d810-43a0-afa2-cf8083bc8874\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " Nov 03 00:16:38.335589 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.335481 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-token\" (UniqueName: \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-token\") pod \"7ed3e348-d810-43a0-afa2-cf8083bc8874\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " Nov 03 00:16:38.335589 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.335565 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed3e348-d810-43a0-afa2-cf8083bc8874-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed3e348-d810-43a0-afa2-cf8083bc8874-istiod-ca-cert\") pod \"7ed3e348-d810-43a0-afa2-cf8083bc8874\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " Nov 03 00:16:38.335758 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.335597 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-kube-api-access-sq89f\" (UniqueName: \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-kube-api-access-sq89f\") pod \"7ed3e348-d810-43a0-afa2-cf8083bc8874\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " Nov 03 00:16:38.335758 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.335641 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-credential-socket\") pod \"7ed3e348-d810-43a0-afa2-cf8083bc8874\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " Nov 03 00:16:38.335758 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.335719 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-envoy\") pod \"7ed3e348-d810-43a0-afa2-cf8083bc8874\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " Nov 03 00:16:38.335932 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.335761 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-socket\") pod \"7ed3e348-d810-43a0-afa2-cf8083bc8874\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " Nov 03 00:16:38.335932 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.335810 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-certs\") pod \"7ed3e348-d810-43a0-afa2-cf8083bc8874\" (UID: \"7ed3e348-d810-43a0-afa2-cf8083bc8874\") " Nov 03 00:16:38.336731 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.336543 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-data" pod "7ed3e348-d810-43a0-afa2-cf8083bc8874" (UID: "7ed3e348-d810-43a0-afa2-cf8083bc8874"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:38.336976 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.336905 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-credential-socket" pod "7ed3e348-d810-43a0-afa2-cf8083bc8874" (UID: "7ed3e348-d810-43a0-afa2-cf8083bc8874"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:38.337235 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.337183 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-socket" pod "7ed3e348-d810-43a0-afa2-cf8083bc8874" (UID: "7ed3e348-d810-43a0-afa2-cf8083bc8874"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:38.337235 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.337225 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed3e348-d810-43a0-afa2-cf8083bc8874-istiod-ca-cert" pod "7ed3e348-d810-43a0-afa2-cf8083bc8874" (UID: "7ed3e348-d810-43a0-afa2-cf8083bc8874"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:38.337859 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.337829 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-certs" pod "7ed3e348-d810-43a0-afa2-cf8083bc8874" (UID: "7ed3e348-d810-43a0-afa2-cf8083bc8874"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:38.338292 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.338272 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:38.338444 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.338428 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:38.338628 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.338611 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:38.338735 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.338721 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:38.338862 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.338841 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed3e348-d810-43a0-afa2-cf8083bc8874-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:38.339076 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.339044 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-token" pod "7ed3e348-d810-43a0-afa2-cf8083bc8874" (UID: "7ed3e348-d810-43a0-afa2-cf8083bc8874"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:38.339610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.339593 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92"] Nov 03 00:16:38.340453 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.340103 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-podinfo" pod "7ed3e348-d810-43a0-afa2-cf8083bc8874" (UID: "7ed3e348-d810-43a0-afa2-cf8083bc8874"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:16:38.341454 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.341417 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-envoy" pod "7ed3e348-d810-43a0-afa2-cf8083bc8874" (UID: "7ed3e348-d810-43a0-afa2-cf8083bc8874"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:38.341557 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.341495 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-kube-api-access-sq89f" pod "7ed3e348-d810-43a0-afa2-cf8083bc8874" (UID: "7ed3e348-d810-43a0-afa2-cf8083bc8874"). InnerVolumeSpecName "kube-api-access-sq89f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:38.344471 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.344447 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq"] Nov 03 00:16:38.439826 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.439743 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:38.439826 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.439773 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:38.439826 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.439783 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:38.439826 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.439793 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq89f\" (UniqueName: \"kubernetes.io/projected/7ed3e348-d810-43a0-afa2-cf8083bc8874-kube-api-access-sq89f\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:38.831777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.831741 5672 generic.go:356] "Generic (PLEG): container finished" podID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerID="2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110" exitCode=0 Nov 03 00:16:38.831983 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.831866 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" Nov 03 00:16:38.831983 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.831890 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" event={"ID":"7ed3e348-d810-43a0-afa2-cf8083bc8874","Type":"ContainerDied","Data":"2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110"} Nov 03 00:16:38.831983 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.831956 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh" event={"ID":"7ed3e348-d810-43a0-afa2-cf8083bc8874","Type":"ContainerDied","Data":"d42154935c322940910e879f16a74cc2278761a2754333c1eed3fa7cff28a346"} Nov 03 00:16:38.832135 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.831985 5672 scope.go:122] "RemoveContainer" containerID="2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110" Nov 03 00:16:38.844304 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.844281 5672 scope.go:122] "RemoveContainer" containerID="ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b" Nov 03 00:16:38.855975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.855950 5672 scope.go:122] "RemoveContainer" containerID="2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110" Nov 03 00:16:38.856337 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:16:38.856314 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110\": container with ID starting with 2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110 not found: ID does not exist" containerID="2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110" Nov 03 00:16:38.856430 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.856350 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110"} err="failed to get container status \"2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110\": rpc error: code = NotFound desc = could not find container \"2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110\": container with ID starting with 2f3264c2d8dd7f6aeaf7c8d34250348fb2661bfb3c9d3cb96bcd5b4d4510f110 not found: ID does not exist" Nov 03 00:16:38.856430 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.856374 5672 scope.go:122] "RemoveContainer" containerID="ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b" Nov 03 00:16:38.856690 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:16:38.856662 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b\": container with ID starting with ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b not found: ID does not exist" containerID="ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b" Nov 03 00:16:38.856762 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.856697 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b"} err="failed to get container status \"ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b\": rpc error: code = NotFound desc = could not find container \"ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b\": container with ID starting with ca77e3a9689a598de42f7117ac30fd028952dd632761a67541f3e72c6e77eb3b not found: ID does not exist" Nov 03 00:16:38.986179 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.986078 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh"] Nov 03 00:16:38.991685 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:38.991654 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fbec82-openshift-default-55bcb7f997-5kllh"] Nov 03 00:16:40.657096 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:40.657047 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" podUID="1efd45fd-b21e-4ca7-b2d7-88f15ae1d168" containerName="payload-processing" containerID="cri-o://bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b" gracePeriod=25 Nov 03 00:16:40.708139 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:40.708092 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" podUID="49e02f1e-bd39-416b-9323-5eed40f53cb3" containerName="payload-pre-processing" containerID="cri-o://5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db" gracePeriod=25 Nov 03 00:16:40.713199 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:40.713172 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" path="/var/lib/kubelet/pods/7ed3e348-d810-43a0-afa2-cf8083bc8874/volumes" Nov 03 00:16:44.660453 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.660412 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9"] Nov 03 00:16:44.660965 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.660870 5672 cpu_manager.go:414] "containerMap: removing container" podUID="424d3364-432e-4dde-ac07-75108c4ca592" containerName="revoke-keys" Nov 03 00:16:44.660965 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.660883 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424d3364-432e-4dde-ac07-75108c4ca592" containerName="revoke-keys" Nov 03 00:16:44.660965 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.660899 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" Nov 03 00:16:44.660965 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.660905 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" Nov 03 00:16:44.660965 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.660918 5672 cpu_manager.go:414] "containerMap: removing container" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" Nov 03 00:16:44.660965 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.660922 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" Nov 03 00:16:44.661337 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.660992 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="424d3364-432e-4dde-ac07-75108c4ca592" containerName="revoke-keys" Nov 03 00:16:44.661337 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.661003 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" Nov 03 00:16:44.661337 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.661010 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed3e348-d810-43a0-afa2-cf8083bc8874" containerName="istio-proxy" Nov 03 00:16:44.665927 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.665903 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:44.704747 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.704713 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ea58fc-v46n9\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:44.704893 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.704771 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fv8\" (UniqueName: \"kubernetes.io/projected/fbe00cb2-b676-42c1-beaf-bce901ce8394-kube-api-access-h7fv8\") pod \"maas-api-revoke-sub-e1ea58fc-v46n9\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:44.704937 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.704898 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ea58fc-v46n9\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:44.716881 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.716851 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9"] Nov 03 00:16:44.759957 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.759922 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:16:44.759957 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.759922 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:16:44.760197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.759971 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:16:44.760878 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.760861 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:16:44.805976 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.805942 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ea58fc-v46n9\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:44.806197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.805993 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fv8\" (UniqueName: \"kubernetes.io/projected/fbe00cb2-b676-42c1-beaf-bce901ce8394-kube-api-access-h7fv8\") pod \"maas-api-revoke-sub-e1ea58fc-v46n9\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:44.806197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.806144 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ea58fc-v46n9\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:44.806713 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.806690 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ea58fc-v46n9\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:44.808481 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.808462 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ea58fc-v46n9\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:44.861131 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.861091 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fv8\" (UniqueName: \"kubernetes.io/projected/fbe00cb2-b676-42c1-beaf-bce901ce8394-kube-api-access-h7fv8\") pod \"maas-api-revoke-sub-e1ea58fc-v46n9\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:44.981168 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:44.981105 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:45.349480 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:45.349445 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9"] Nov 03 00:16:45.351387 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:16:45.351359 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe00cb2_b676_42c1_beaf_bce901ce8394.slice/crio-d1f213fe285471cbf4d1ee8beda537474fbd9532bff49e3f43c6747a26548529 WatchSource:0}: Error finding container d1f213fe285471cbf4d1ee8beda537474fbd9532bff49e3f43c6747a26548529: Status 404 returned error can't find the container with id d1f213fe285471cbf4d1ee8beda537474fbd9532bff49e3f43c6747a26548529 Nov 03 00:16:45.892897 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:45.892779 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" event={"ID":"fbe00cb2-b676-42c1-beaf-bce901ce8394","Type":"ContainerStarted","Data":"0d2dd95cf25e00dc3f9a8a154970a54294ecec79d3800479cc2c168cd3685006"} Nov 03 00:16:45.892897 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:45.892846 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" event={"ID":"fbe00cb2-b676-42c1-beaf-bce901ce8394","Type":"ContainerStarted","Data":"d1f213fe285471cbf4d1ee8beda537474fbd9532bff49e3f43c6747a26548529"} Nov 03 00:16:45.999904 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:45.999847 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" podStartSLOduration=1.9998295719999999 podStartE2EDuration="1.999829572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:45.998138641 +0000 UTC m=+4999.405450506" watchObservedRunningTime="2026-11-03 00:16:45.999829572 +0000 UTC m=+4999.407141401" Nov 03 00:16:46.801772 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:46.801731 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" podUID="1efd45fd-b21e-4ca7-b2d7-88f15ae1d168" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:16:46.801953 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:46.801776 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" podUID="49e02f1e-bd39-416b-9323-5eed40f53cb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:16:46.900631 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:46.900595 5672 generic.go:356] "Generic (PLEG): container finished" podID="fbe00cb2-b676-42c1-beaf-bce901ce8394" containerID="0d2dd95cf25e00dc3f9a8a154970a54294ecec79d3800479cc2c168cd3685006" exitCode=0 Nov 03 00:16:46.901022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:46.900636 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" event={"ID":"fbe00cb2-b676-42c1-beaf-bce901ce8394","Type":"ContainerDied","Data":"0d2dd95cf25e00dc3f9a8a154970a54294ecec79d3800479cc2c168cd3685006"} Nov 03 00:16:48.075942 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.075918 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:48.141960 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.141918 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-openshift-ca\") pod \"fbe00cb2-b676-42c1-beaf-bce901ce8394\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " Nov 03 00:16:48.142183 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.142094 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe00cb2-b676-42c1-beaf-bce901ce8394-kube-api-access-h7fv8\" (UniqueName: \"kubernetes.io/projected/fbe00cb2-b676-42c1-beaf-bce901ce8394-kube-api-access-h7fv8\") pod \"fbe00cb2-b676-42c1-beaf-bce901ce8394\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " Nov 03 00:16:48.142183 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.142138 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-xks-ca\") pod \"fbe00cb2-b676-42c1-beaf-bce901ce8394\" (UID: \"fbe00cb2-b676-42c1-beaf-bce901ce8394\") " Nov 03 00:16:48.142444 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.142413 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-openshift-ca" pod "fbe00cb2-b676-42c1-beaf-bce901ce8394" (UID: "fbe00cb2-b676-42c1-beaf-bce901ce8394"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:48.142606 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.142586 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:48.144284 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.144256 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe00cb2-b676-42c1-beaf-bce901ce8394-kube-api-access-h7fv8" pod "fbe00cb2-b676-42c1-beaf-bce901ce8394" (UID: "fbe00cb2-b676-42c1-beaf-bce901ce8394"). InnerVolumeSpecName "kube-api-access-h7fv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:48.144453 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.144431 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-xks-ca" pod "fbe00cb2-b676-42c1-beaf-bce901ce8394" (UID: "fbe00cb2-b676-42c1-beaf-bce901ce8394"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:48.244115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.244069 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7fv8\" (UniqueName: \"kubernetes.io/projected/fbe00cb2-b676-42c1-beaf-bce901ce8394-kube-api-access-h7fv8\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:48.244115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.244108 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe00cb2-b676-42c1-beaf-bce901ce8394-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:48.917046 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.917015 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" Nov 03 00:16:48.917250 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.917010 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9" event={"ID":"fbe00cb2-b676-42c1-beaf-bce901ce8394","Type":"ContainerDied","Data":"d1f213fe285471cbf4d1ee8beda537474fbd9532bff49e3f43c6747a26548529"} Nov 03 00:16:48.917250 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:48.917133 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1f213fe285471cbf4d1ee8beda537474fbd9532bff49e3f43c6747a26548529" Nov 03 00:16:49.017190 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.017087 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd"] Nov 03 00:16:49.019870 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.019833 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q"] Nov 03 00:16:49.719681 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.719649 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n"] Nov 03 00:16:49.720105 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.720092 5672 cpu_manager.go:414] "containerMap: removing container" podUID="fbe00cb2-b676-42c1-beaf-bce901ce8394" containerName="revoke-keys" Nov 03 00:16:49.720169 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.720105 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe00cb2-b676-42c1-beaf-bce901ce8394" containerName="revoke-keys" Nov 03 00:16:49.720238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.720228 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe00cb2-b676-42c1-beaf-bce901ce8394" containerName="revoke-keys" Nov 03 00:16:49.723686 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.723663 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:49.760317 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.760278 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da84f27e-wwn8n\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:49.760492 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.760417 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfkdz\" (UniqueName: \"kubernetes.io/projected/bf7f2b93-fd95-4de4-ba73-7971096e0971-kube-api-access-wfkdz\") pod \"maas-api-revoke-sub-da84f27e-wwn8n\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:49.760663 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.760638 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da84f27e-wwn8n\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:49.773325 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.773294 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:16:49.773325 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.773311 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:16:49.777335 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.777304 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n"] Nov 03 00:16:49.818740 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.818707 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:16:49.818740 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.818717 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:16:49.861605 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.861568 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfkdz\" (UniqueName: \"kubernetes.io/projected/bf7f2b93-fd95-4de4-ba73-7971096e0971-kube-api-access-wfkdz\") pod \"maas-api-revoke-sub-da84f27e-wwn8n\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:49.861782 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.861653 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da84f27e-wwn8n\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:49.861852 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.861794 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da84f27e-wwn8n\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:49.862505 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.862480 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da84f27e-wwn8n\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:49.864369 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.864347 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da84f27e-wwn8n\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:49.914925 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:49.914894 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfkdz\" (UniqueName: \"kubernetes.io/projected/bf7f2b93-fd95-4de4-ba73-7971096e0971-kube-api-access-wfkdz\") pod \"maas-api-revoke-sub-da84f27e-wwn8n\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:50.041319 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:50.041234 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:50.416798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:50.416770 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n"] Nov 03 00:16:50.418228 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:16:50.418189 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7f2b93_fd95_4de4_ba73_7971096e0971.slice/crio-b15b6347d1aa9538b384d014db4380a324164a91bfd273a6d08a0b66e13aef18 WatchSource:0}: Error finding container b15b6347d1aa9538b384d014db4380a324164a91bfd273a6d08a0b66e13aef18: Status 404 returned error can't find the container with id b15b6347d1aa9538b384d014db4380a324164a91bfd273a6d08a0b66e13aef18 Nov 03 00:16:50.937093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:50.937049 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" event={"ID":"bf7f2b93-fd95-4de4-ba73-7971096e0971","Type":"ContainerStarted","Data":"739dcacaadfac9e964fffb3704f79192f817f33bff79c7c38ed1103a5860d55c"} Nov 03 00:16:50.937591 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:50.937119 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" event={"ID":"bf7f2b93-fd95-4de4-ba73-7971096e0971","Type":"ContainerStarted","Data":"b15b6347d1aa9538b384d014db4380a324164a91bfd273a6d08a0b66e13aef18"} Nov 03 00:16:51.045640 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:51.045527 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" podStartSLOduration=2.045506122 podStartE2EDuration="2.045506122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:51.044629541 +0000 UTC m=+5004.451941375" watchObservedRunningTime="2026-11-03 00:16:51.045506122 +0000 UTC m=+5004.452817952" Nov 03 00:16:51.945899 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:51.945861 5672 generic.go:356] "Generic (PLEG): container finished" podID="bf7f2b93-fd95-4de4-ba73-7971096e0971" containerID="739dcacaadfac9e964fffb3704f79192f817f33bff79c7c38ed1103a5860d55c" exitCode=0 Nov 03 00:16:51.946444 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:51.945943 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" event={"ID":"bf7f2b93-fd95-4de4-ba73-7971096e0971","Type":"ContainerDied","Data":"739dcacaadfac9e964fffb3704f79192f817f33bff79c7c38ed1103a5860d55c"} Nov 03 00:16:53.105507 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.105478 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:53.207033 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.206933 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-xks-ca\") pod \"bf7f2b93-fd95-4de4-ba73-7971096e0971\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " Nov 03 00:16:53.207270 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.207082 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-openshift-ca\") pod \"bf7f2b93-fd95-4de4-ba73-7971096e0971\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " Nov 03 00:16:53.207270 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.207102 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7f2b93-fd95-4de4-ba73-7971096e0971-kube-api-access-wfkdz\" (UniqueName: \"kubernetes.io/projected/bf7f2b93-fd95-4de4-ba73-7971096e0971-kube-api-access-wfkdz\") pod \"bf7f2b93-fd95-4de4-ba73-7971096e0971\" (UID: \"bf7f2b93-fd95-4de4-ba73-7971096e0971\") " Nov 03 00:16:53.207503 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.207475 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-openshift-ca" pod "bf7f2b93-fd95-4de4-ba73-7971096e0971" (UID: "bf7f2b93-fd95-4de4-ba73-7971096e0971"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:53.209019 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.208997 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-xks-ca" pod "bf7f2b93-fd95-4de4-ba73-7971096e0971" (UID: "bf7f2b93-fd95-4de4-ba73-7971096e0971"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:53.209313 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.209296 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7f2b93-fd95-4de4-ba73-7971096e0971-kube-api-access-wfkdz" pod "bf7f2b93-fd95-4de4-ba73-7971096e0971" (UID: "bf7f2b93-fd95-4de4-ba73-7971096e0971"). InnerVolumeSpecName "kube-api-access-wfkdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:53.308906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.308871 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.308906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.308900 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfkdz\" (UniqueName: \"kubernetes.io/projected/bf7f2b93-fd95-4de4-ba73-7971096e0971-kube-api-access-wfkdz\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.308906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.308910 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7f2b93-fd95-4de4-ba73-7971096e0971-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.361890 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.361848 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" containerID="cri-o://79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b" gracePeriod=45 Nov 03 00:16:53.366320 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.366120 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" containerID="cri-o://f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b" gracePeriod=45 Nov 03 00:16:53.366726 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.366654 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" containerID="cri-o://b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708" gracePeriod=45 Nov 03 00:16:53.415725 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.415685 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": dial tcp 10.134.0.46:8000: connect: connection refused" Nov 03 00:16:53.418049 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.418020 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 03 00:16:53.614044 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.614016 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 03 00:16:53.670531 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.670500 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 03 00:16:53.681247 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.681221 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 03 00:16:53.711885 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.711855 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-home\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-home\") pod \"60e279ff-eba6-4783-ac4c-dbc824466aa2\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " Nov 03 00:16:53.712045 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.711893 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-tmp-dir\") pod \"60e279ff-eba6-4783-ac4c-dbc824466aa2\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " Nov 03 00:16:53.712045 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.711953 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-model-cache\") pod \"60e279ff-eba6-4783-ac4c-dbc824466aa2\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " Nov 03 00:16:53.712197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.712054 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-dshm\") pod \"60e279ff-eba6-4783-ac4c-dbc824466aa2\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " Nov 03 00:16:53.712197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.712110 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e279ff-eba6-4783-ac4c-dbc824466aa2-tls-certs\" (UniqueName: \"kubernetes.io/secret/60e279ff-eba6-4783-ac4c-dbc824466aa2-tls-certs\") pod \"60e279ff-eba6-4783-ac4c-dbc824466aa2\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " Nov 03 00:16:53.712197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.712124 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-home" pod "60e279ff-eba6-4783-ac4c-dbc824466aa2" (UID: "60e279ff-eba6-4783-ac4c-dbc824466aa2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.712197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.712162 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e279ff-eba6-4783-ac4c-dbc824466aa2-kube-api-access-fbf7v\" (UniqueName: \"kubernetes.io/projected/60e279ff-eba6-4783-ac4c-dbc824466aa2-kube-api-access-fbf7v\") pod \"60e279ff-eba6-4783-ac4c-dbc824466aa2\" (UID: \"60e279ff-eba6-4783-ac4c-dbc824466aa2\") " Nov 03 00:16:53.712431 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.712221 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-tmp-dir" pod "60e279ff-eba6-4783-ac4c-dbc824466aa2" (UID: "60e279ff-eba6-4783-ac4c-dbc824466aa2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.712538 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.712516 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-model-cache" pod "60e279ff-eba6-4783-ac4c-dbc824466aa2" (UID: "60e279ff-eba6-4783-ac4c-dbc824466aa2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.712603 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.712587 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.712660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.712608 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.714473 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.714424 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e279ff-eba6-4783-ac4c-dbc824466aa2-kube-api-access-fbf7v" pod "60e279ff-eba6-4783-ac4c-dbc824466aa2" (UID: "60e279ff-eba6-4783-ac4c-dbc824466aa2"). InnerVolumeSpecName "kube-api-access-fbf7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:53.714625 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.714600 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-dshm" pod "60e279ff-eba6-4783-ac4c-dbc824466aa2" (UID: "60e279ff-eba6-4783-ac4c-dbc824466aa2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.715061 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.715041 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e279ff-eba6-4783-ac4c-dbc824466aa2-tls-certs" pod "60e279ff-eba6-4783-ac4c-dbc824466aa2" (UID: "60e279ff-eba6-4783-ac4c-dbc824466aa2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:53.813093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813003 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-model-cache\") pod \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " Nov 03 00:16:53.813093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813056 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-tmp-dir\") pod \"4278c7e1-9584-47f4-ac3d-323867b57b83\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " Nov 03 00:16:53.813353 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813106 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-kube-api-access-42dpf\" (UniqueName: \"kubernetes.io/projected/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-kube-api-access-42dpf\") pod \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " Nov 03 00:16:53.813353 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813203 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-model-cache\") pod \"4278c7e1-9584-47f4-ac3d-323867b57b83\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " Nov 03 00:16:53.813353 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813237 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tls-certs\" (UniqueName: \"kubernetes.io/secret/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tls-certs\") pod \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " Nov 03 00:16:53.813353 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813311 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-dshm\") pod \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " Nov 03 00:16:53.813353 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813337 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-dshm\") pod \"4278c7e1-9584-47f4-ac3d-323867b57b83\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " Nov 03 00:16:53.813633 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813405 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-home\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-home\") pod \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " Nov 03 00:16:53.813633 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813434 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4278c7e1-9584-47f4-ac3d-323867b57b83-kube-api-access-hkttz\" (UniqueName: \"kubernetes.io/projected/4278c7e1-9584-47f4-ac3d-323867b57b83-kube-api-access-hkttz\") pod \"4278c7e1-9584-47f4-ac3d-323867b57b83\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " Nov 03 00:16:53.813633 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813462 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4278c7e1-9584-47f4-ac3d-323867b57b83-tls-certs\" (UniqueName: \"kubernetes.io/secret/4278c7e1-9584-47f4-ac3d-323867b57b83-tls-certs\") pod \"4278c7e1-9584-47f4-ac3d-323867b57b83\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " Nov 03 00:16:53.813633 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813497 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tmp-dir\") pod \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\" (UID: \"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d\") " Nov 03 00:16:53.813633 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813567 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-home\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-home\") pod \"4278c7e1-9584-47f4-ac3d-323867b57b83\" (UID: \"4278c7e1-9584-47f4-ac3d-323867b57b83\") " Nov 03 00:16:53.814100 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.814075 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.814215 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.814104 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e279ff-eba6-4783-ac4c-dbc824466aa2-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.814215 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.814120 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e279ff-eba6-4783-ac4c-dbc824466aa2-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.814215 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.814136 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbf7v\" (UniqueName: \"kubernetes.io/projected/60e279ff-eba6-4783-ac4c-dbc824466aa2-kube-api-access-fbf7v\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.815289 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813305 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-model-cache" pod "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" (UID: "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.815289 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813317 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-tmp-dir" pod "4278c7e1-9584-47f4-ac3d-323867b57b83" (UID: "4278c7e1-9584-47f4-ac3d-323867b57b83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.815461 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.813509 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-model-cache" pod "4278c7e1-9584-47f4-ac3d-323867b57b83" (UID: "4278c7e1-9584-47f4-ac3d-323867b57b83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.815461 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.814045 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-home" pod "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" (UID: "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.815461 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.814506 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tmp-dir" pod "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" (UID: "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.815461 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.814682 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-home" pod "4278c7e1-9584-47f4-ac3d-323867b57b83" (UID: "4278c7e1-9584-47f4-ac3d-323867b57b83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.817473 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.817330 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tls-certs" pod "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" (UID: "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:53.817473 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.817411 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-dshm" pod "4278c7e1-9584-47f4-ac3d-323867b57b83" (UID: "4278c7e1-9584-47f4-ac3d-323867b57b83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.817473 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.817434 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-kube-api-access-42dpf" pod "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" (UID: "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d"). InnerVolumeSpecName "kube-api-access-42dpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:53.817704 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.817490 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-dshm" pod "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" (UID: "610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:53.817704 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.817631 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4278c7e1-9584-47f4-ac3d-323867b57b83-kube-api-access-hkttz" pod "4278c7e1-9584-47f4-ac3d-323867b57b83" (UID: "4278c7e1-9584-47f4-ac3d-323867b57b83"). InnerVolumeSpecName "kube-api-access-hkttz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:53.817891 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.817867 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4278c7e1-9584-47f4-ac3d-323867b57b83-tls-certs" pod "4278c7e1-9584-47f4-ac3d-323867b57b83" (UID: "4278c7e1-9584-47f4-ac3d-323867b57b83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:16:53.915011 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.914977 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.915011 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.915013 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.915246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.915027 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.915246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.915042 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkttz\" (UniqueName: \"kubernetes.io/projected/4278c7e1-9584-47f4-ac3d-323867b57b83-kube-api-access-hkttz\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.915246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.915058 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4278c7e1-9584-47f4-ac3d-323867b57b83-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.915246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.915071 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.915246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.915085 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.915246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.915098 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.915246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.915112 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.915246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.915125 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42dpf\" (UniqueName: \"kubernetes.io/projected/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-kube-api-access-42dpf\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.915246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.915139 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4278c7e1-9584-47f4-ac3d-323867b57b83-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.915246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.915185 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:16:53.963078 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.963029 5672 generic.go:356] "Generic (PLEG): container finished" podID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerID="f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b" exitCode=0 Nov 03 00:16:53.963398 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.963092 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" Nov 03 00:16:53.963398 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.963282 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" event={"ID":"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d","Type":"ContainerDied","Data":"f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b"} Nov 03 00:16:53.963398 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.963325 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq" event={"ID":"610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d","Type":"ContainerDied","Data":"663c037ca9ad29fbd8444b0ccca84bae5a48b2d9b495a0ce909cd09956667c6c"} Nov 03 00:16:53.963398 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.963349 5672 scope.go:122] "RemoveContainer" containerID="f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b" Nov 03 00:16:53.967952 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.967923 5672 generic.go:356] "Generic (PLEG): container finished" podID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerID="79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b" exitCode=0 Nov 03 00:16:53.970216 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.968014 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" event={"ID":"60e279ff-eba6-4783-ac4c-dbc824466aa2","Type":"ContainerDied","Data":"79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b"} Nov 03 00:16:53.970216 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.968048 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" event={"ID":"60e279ff-eba6-4783-ac4c-dbc824466aa2","Type":"ContainerDied","Data":"1883490c10322f707aa71f16886688424ea66efa66f6375e4d1e2b439aa64584"} Nov 03 00:16:53.970216 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.968122 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" Nov 03 00:16:53.974678 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.974654 5672 generic.go:356] "Generic (PLEG): container finished" podID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerID="b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708" exitCode=0 Nov 03 00:16:53.974797 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.974770 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" event={"ID":"4278c7e1-9584-47f4-ac3d-323867b57b83","Type":"ContainerDied","Data":"b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708"} Nov 03 00:16:53.974921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.974799 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" event={"ID":"4278c7e1-9584-47f4-ac3d-323867b57b83","Type":"ContainerDied","Data":"f12e99dd9debd1125606df86e4d4296072ba0221fa57fb6460e222510d15fcce"} Nov 03 00:16:53.974921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.974862 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb" Nov 03 00:16:53.987352 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.981978 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" event={"ID":"bf7f2b93-fd95-4de4-ba73-7971096e0971","Type":"ContainerDied","Data":"b15b6347d1aa9538b384d014db4380a324164a91bfd273a6d08a0b66e13aef18"} Nov 03 00:16:53.987352 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.982026 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b15b6347d1aa9538b384d014db4380a324164a91bfd273a6d08a0b66e13aef18" Nov 03 00:16:53.987352 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.982088 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n" Nov 03 00:16:53.989575 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.989555 5672 scope.go:122] "RemoveContainer" containerID="f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b" Nov 03 00:16:53.990176 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:16:53.990119 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b\": container with ID starting with f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b not found: ID does not exist" containerID="f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b" Nov 03 00:16:53.990545 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.990517 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b"} err="failed to get container status \"f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b\": rpc error: code = NotFound desc = could not find container \"f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b\": container with ID starting with f6c66448ebc3a3f924df9122b04377dcee303b4d2cd41bf0b250f50124643f7b not found: ID does not exist" Nov 03 00:16:53.990634 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:53.990549 5672 scope.go:122] "RemoveContainer" containerID="79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b" Nov 03 00:16:54.003786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.003760 5672 scope.go:122] "RemoveContainer" containerID="79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b" Nov 03 00:16:54.004179 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:16:54.004120 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b\": container with ID starting with 79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b not found: ID does not exist" containerID="79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b" Nov 03 00:16:54.004352 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.004316 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b"} err="failed to get container status \"79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b\": rpc error: code = NotFound desc = could not find container \"79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b\": container with ID starting with 79587bbf859bb1b8eccd60d805d3dfcd462afa6094c4005e2346f38065d8440b not found: ID does not exist" Nov 03 00:16:54.004455 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.004354 5672 scope.go:122] "RemoveContainer" containerID="b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708" Nov 03 00:16:54.014291 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.014271 5672 scope.go:122] "RemoveContainer" containerID="b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708" Nov 03 00:16:54.015343 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:16:54.015311 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708\": container with ID starting with b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708 not found: ID does not exist" containerID="b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708" Nov 03 00:16:54.015458 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.015349 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708"} err="failed to get container status \"b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708\": rpc error: code = NotFound desc = could not find container \"b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708\": container with ID starting with b4ab224414df07ed68ea25d82968b70f5b9731c60b00b8ae4c6b439305009708 not found: ID does not exist" Nov 03 00:16:54.116902 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.116870 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb"] Nov 03 00:16:54.120445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.120417 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-2-w2-ba1918-kserve-66cd4ddcd-26rbb"] Nov 03 00:16:54.223358 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.223323 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92"] Nov 03 00:16:54.226072 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.226047 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92"] Nov 03 00:16:54.327405 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.327368 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq"] Nov 03 00:16:54.333074 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.333001 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/simulator-w2-ba1918-kserve-65ddbfc79f-4x5jq"] Nov 03 00:16:54.579844 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.579802 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/distinct-w2-ba1918-kserve-75bcd4968f-phg92" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:16:54.712237 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.712166 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" path="/var/lib/kubelet/pods/4278c7e1-9584-47f4-ac3d-323867b57b83/volumes" Nov 03 00:16:54.712658 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.712644 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" path="/var/lib/kubelet/pods/60e279ff-eba6-4783-ac4c-dbc824466aa2/volumes" Nov 03 00:16:54.713054 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:54.713042 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" path="/var/lib/kubelet/pods/610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d/volumes" Nov 03 00:16:56.021875 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:16:56.021836 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:16:56.801311 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:56.801275 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" podUID="1efd45fd-b21e-4ca7-b2d7-88f15ae1d168" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:16:56.801490 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:16:56.801276 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" podUID="49e02f1e-bd39-416b-9323-5eed40f53cb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:17:00.982201 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:00.982173 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 03 00:17:01.011243 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.011220 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 03 00:17:01.033121 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.033087 5672 generic.go:356] "Generic (PLEG): container finished" podID="49e02f1e-bd39-416b-9323-5eed40f53cb3" containerID="5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db" exitCode=0 Nov 03 00:17:01.033399 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.033134 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" Nov 03 00:17:01.033399 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.033175 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" event={"ID":"49e02f1e-bd39-416b-9323-5eed40f53cb3","Type":"ContainerDied","Data":"5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db"} Nov 03 00:17:01.033399 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.033227 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj" event={"ID":"49e02f1e-bd39-416b-9323-5eed40f53cb3","Type":"ContainerDied","Data":"d2cdf8d33bbe636dd19933c4394f485e7cc2b63bf9d2953696dfe5f43930b68c"} Nov 03 00:17:01.033399 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.033251 5672 scope.go:122] "RemoveContainer" containerID="5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db" Nov 03 00:17:01.034855 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.034831 5672 generic.go:356] "Generic (PLEG): container finished" podID="1efd45fd-b21e-4ca7-b2d7-88f15ae1d168" containerID="bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b" exitCode=0 Nov 03 00:17:01.034961 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.034883 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" Nov 03 00:17:01.034961 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.034916 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" event={"ID":"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168","Type":"ContainerDied","Data":"bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b"} Nov 03 00:17:01.035059 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.034967 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw" event={"ID":"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168","Type":"ContainerDied","Data":"116c53299656899ab36d1fba69dbf19484bbcfacdaa933da97ad5b76d8a74b4f"} Nov 03 00:17:01.050942 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.050837 5672 scope.go:122] "RemoveContainer" containerID="5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db" Nov 03 00:17:01.052266 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:17:01.052076 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db\": container with ID starting with 5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db not found: ID does not exist" containerID="5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db" Nov 03 00:17:01.052266 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.052117 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db"} err="failed to get container status \"5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db\": rpc error: code = NotFound desc = could not find container \"5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db\": container with ID starting with 5174c2b41e4b5da8b577c4b0ad2eaca0c47b13512bb0c79df566166c40b094db not found: ID does not exist" Nov 03 00:17:01.052266 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.052140 5672 scope.go:122] "RemoveContainer" containerID="bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b" Nov 03 00:17:01.068252 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.068227 5672 scope.go:122] "RemoveContainer" containerID="bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b" Nov 03 00:17:01.068610 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:17:01.068585 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b\": container with ID starting with bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b not found: ID does not exist" containerID="bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b" Nov 03 00:17:01.068686 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.068623 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b"} err="failed to get container status \"bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b\": rpc error: code = NotFound desc = could not find container \"bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b\": container with ID starting with bc7c01e2db333c3c8d36bf378e8736e5011bf1c8a0b5322bddc3e19f99ffaa0b not found: ID does not exist" Nov 03 00:17:01.092972 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.092942 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-kube-api-access-q85zc\" (UniqueName: \"kubernetes.io/projected/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-kube-api-access-q85zc\") pod \"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168\" (UID: \"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168\") " Nov 03 00:17:01.093120 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.092984 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-plugins-config-volume\") pod \"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168\" (UID: \"1efd45fd-b21e-4ca7-b2d7-88f15ae1d168\") " Nov 03 00:17:01.093120 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.093100 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e02f1e-bd39-416b-9323-5eed40f53cb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e02f1e-bd39-416b-9323-5eed40f53cb3-plugins-config-volume\") pod \"49e02f1e-bd39-416b-9323-5eed40f53cb3\" (UID: \"49e02f1e-bd39-416b-9323-5eed40f53cb3\") " Nov 03 00:17:01.093280 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.093240 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e02f1e-bd39-416b-9323-5eed40f53cb3-kube-api-access-tgm82\" (UniqueName: \"kubernetes.io/projected/49e02f1e-bd39-416b-9323-5eed40f53cb3-kube-api-access-tgm82\") pod \"49e02f1e-bd39-416b-9323-5eed40f53cb3\" (UID: \"49e02f1e-bd39-416b-9323-5eed40f53cb3\") " Nov 03 00:17:01.093427 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.093400 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-plugins-config-volume" pod "1efd45fd-b21e-4ca7-b2d7-88f15ae1d168" (UID: "1efd45fd-b21e-4ca7-b2d7-88f15ae1d168"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:01.093512 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.093472 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e02f1e-bd39-416b-9323-5eed40f53cb3-plugins-config-volume" pod "49e02f1e-bd39-416b-9323-5eed40f53cb3" (UID: "49e02f1e-bd39-416b-9323-5eed40f53cb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:01.093598 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.093582 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:01.093660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.093600 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e02f1e-bd39-416b-9323-5eed40f53cb3-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:01.095268 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.095240 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-kube-api-access-q85zc" pod "1efd45fd-b21e-4ca7-b2d7-88f15ae1d168" (UID: "1efd45fd-b21e-4ca7-b2d7-88f15ae1d168"). InnerVolumeSpecName "kube-api-access-q85zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:01.095403 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.095382 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e02f1e-bd39-416b-9323-5eed40f53cb3-kube-api-access-tgm82" pod "49e02f1e-bd39-416b-9323-5eed40f53cb3" (UID: "49e02f1e-bd39-416b-9323-5eed40f53cb3"). InnerVolumeSpecName "kube-api-access-tgm82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:01.176989 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.176896 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:17:01.194854 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.194825 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q85zc\" (UniqueName: \"kubernetes.io/projected/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168-kube-api-access-q85zc\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:01.194854 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.194848 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgm82\" (UniqueName: \"kubernetes.io/projected/49e02f1e-bd39-416b-9323-5eed40f53cb3-kube-api-access-tgm82\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:01.452455 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.452372 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw"] Nov 03 00:17:01.457232 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.457198 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c25940-5968b7b5d7-whlkw"] Nov 03 00:17:01.559859 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.559826 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj"] Nov 03 00:17:01.564254 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:01.564217 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c25940-67868684d7-l2wjj"] Nov 03 00:17:02.711738 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.711699 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1efd45fd-b21e-4ca7-b2d7-88f15ae1d168" path="/var/lib/kubelet/pods/1efd45fd-b21e-4ca7-b2d7-88f15ae1d168/volumes" Nov 03 00:17:02.712224 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.712206 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e02f1e-bd39-416b-9323-5eed40f53cb3" path="/var/lib/kubelet/pods/49e02f1e-bd39-416b-9323-5eed40f53cb3/volumes" Nov 03 00:17:02.815166 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815128 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt"] Nov 03 00:17:02.815598 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815584 5672 cpu_manager.go:414] "containerMap: removing container" podUID="1efd45fd-b21e-4ca7-b2d7-88f15ae1d168" containerName="payload-processing" Nov 03 00:17:02.815641 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815600 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efd45fd-b21e-4ca7-b2d7-88f15ae1d168" containerName="payload-processing" Nov 03 00:17:02.815641 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815620 5672 cpu_manager.go:414] "containerMap: removing container" podUID="bf7f2b93-fd95-4de4-ba73-7971096e0971" containerName="revoke-keys" Nov 03 00:17:02.815641 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815629 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7f2b93-fd95-4de4-ba73-7971096e0971" containerName="revoke-keys" Nov 03 00:17:02.815745 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815647 5672 cpu_manager.go:414] "containerMap: removing container" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" Nov 03 00:17:02.815745 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815653 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" Nov 03 00:17:02.815745 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815665 5672 cpu_manager.go:414] "containerMap: removing container" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" Nov 03 00:17:02.815745 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815670 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" Nov 03 00:17:02.815745 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815682 5672 cpu_manager.go:414] "containerMap: removing container" podUID="49e02f1e-bd39-416b-9323-5eed40f53cb3" containerName="payload-pre-processing" Nov 03 00:17:02.815745 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815687 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e02f1e-bd39-416b-9323-5eed40f53cb3" containerName="payload-pre-processing" Nov 03 00:17:02.815745 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815697 5672 cpu_manager.go:414] "containerMap: removing container" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" Nov 03 00:17:02.815745 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815705 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" Nov 03 00:17:02.816004 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815787 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e02f1e-bd39-416b-9323-5eed40f53cb3" containerName="payload-pre-processing" Nov 03 00:17:02.816004 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815799 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="610e4cbd-8e1f-4af7-8ef0-f7ecf7b0162d" containerName="main" Nov 03 00:17:02.816004 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815807 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efd45fd-b21e-4ca7-b2d7-88f15ae1d168" containerName="payload-processing" Nov 03 00:17:02.816004 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815813 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7f2b93-fd95-4de4-ba73-7971096e0971" containerName="revoke-keys" Nov 03 00:17:02.816004 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815820 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="4278c7e1-9584-47f4-ac3d-323867b57b83" containerName="main" Nov 03 00:17:02.816004 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.815826 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e279ff-eba6-4783-ac4c-dbc824466aa2" containerName="main" Nov 03 00:17:02.820323 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.820302 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:02.870878 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.870840 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt"] Nov 03 00:17:02.913798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.913766 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49634525-59xtt\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:02.914001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.913819 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49634525-59xtt\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:02.914001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.913913 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmnf\" (UniqueName: \"kubernetes.io/projected/a242d460-6053-4ead-8930-bc07066a2e97-kube-api-access-7fmnf\") pod \"maas-api-revoke-sub-49634525-59xtt\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:02.914315 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.914294 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:17:02.914377 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.914323 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:17:02.917988 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.917965 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:17:02.917988 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:02.917968 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:17:03.015261 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:03.015230 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49634525-59xtt\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:03.015447 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:03.015270 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmnf\" (UniqueName: \"kubernetes.io/projected/a242d460-6053-4ead-8930-bc07066a2e97-kube-api-access-7fmnf\") pod \"maas-api-revoke-sub-49634525-59xtt\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:03.015447 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:03.015371 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49634525-59xtt\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:03.015859 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:03.015838 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49634525-59xtt\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:03.017516 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:03.017493 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49634525-59xtt\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:03.042246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:03.042218 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q"] Nov 03 00:17:03.047983 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:03.047961 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1015f300-hbz6q"] Nov 03 00:17:03.068332 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:03.068306 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmnf\" (UniqueName: \"kubernetes.io/projected/a242d460-6053-4ead-8930-bc07066a2e97-kube-api-access-7fmnf\") pod \"maas-api-revoke-sub-49634525-59xtt\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:03.137490 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:03.137457 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:03.507873 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:03.507846 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt"] Nov 03 00:17:03.508741 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:17:03.508708 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda242d460_6053_4ead_8930_bc07066a2e97.slice/crio-19237f91b95e34ac30b09a7dafbaeed693f39aebb2a4b34f358d0bf452c5ed75 WatchSource:0}: Error finding container 19237f91b95e34ac30b09a7dafbaeed693f39aebb2a4b34f358d0bf452c5ed75: Status 404 returned error can't find the container with id 19237f91b95e34ac30b09a7dafbaeed693f39aebb2a4b34f358d0bf452c5ed75 Nov 03 00:17:04.038411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.038351 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" containerID="cri-o://3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0" gracePeriod=45 Nov 03 00:17:04.039630 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.039594 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" containerID="cri-o://cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993" gracePeriod=45 Nov 03 00:17:04.066226 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.066189 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" event={"ID":"a242d460-6053-4ead-8930-bc07066a2e97","Type":"ContainerStarted","Data":"4a571eb3d7bf81ac9bcb5f13051a51a7a0924c763e83c083bffbba9f6beb12b0"} Nov 03 00:17:04.066364 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.066256 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" event={"ID":"a242d460-6053-4ead-8930-bc07066a2e97","Type":"ContainerStarted","Data":"19237f91b95e34ac30b09a7dafbaeed693f39aebb2a4b34f358d0bf452c5ed75"} Nov 03 00:17:04.174802 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.174749 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" podStartSLOduration=2.174734751 podStartE2EDuration="2.174734751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:04.173307719 +0000 UTC m=+5017.580619552" watchObservedRunningTime="2026-11-03 00:17:04.174734751 +0000 UTC m=+5017.582046578" Nov 03 00:17:04.303377 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.303348 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 03 00:17:04.306400 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.306378 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 03 00:17:04.429728 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.429641 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-model-cache\") pod \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " Nov 03 00:17:04.429728 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.429691 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d891d08-d471-4358-9dea-d25276081c2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d891d08-d471-4358-9dea-d25276081c2f-tls-certs\") pod \"3d891d08-d471-4358-9dea-d25276081c2f\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " Nov 03 00:17:04.429976 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.429733 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe36396-633b-4974-9fdb-16cf3b7f62c7-kube-api-access-z6dgj\" (UniqueName: \"kubernetes.io/projected/1fe36396-633b-4974-9fdb-16cf3b7f62c7-kube-api-access-z6dgj\") pod \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " Nov 03 00:17:04.429976 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.429789 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-home\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-home\") pod \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " Nov 03 00:17:04.430101 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.429949 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-model-cache" pod "1fe36396-633b-4974-9fdb-16cf3b7f62c7" (UID: "1fe36396-633b-4974-9fdb-16cf3b7f62c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:04.430101 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430038 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-home" pod "1fe36396-633b-4974-9fdb-16cf3b7f62c7" (UID: "1fe36396-633b-4974-9fdb-16cf3b7f62c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:04.430101 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430041 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d891d08-d471-4358-9dea-d25276081c2f-kube-api-access-q685s\" (UniqueName: \"kubernetes.io/projected/3d891d08-d471-4358-9dea-d25276081c2f-kube-api-access-q685s\") pod \"3d891d08-d471-4358-9dea-d25276081c2f\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " Nov 03 00:17:04.430303 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430129 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-model-cache\") pod \"3d891d08-d471-4358-9dea-d25276081c2f\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " Nov 03 00:17:04.430369 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430348 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-dshm\") pod \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " Nov 03 00:17:04.430369 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430355 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-model-cache" pod "3d891d08-d471-4358-9dea-d25276081c2f" (UID: "3d891d08-d471-4358-9dea-d25276081c2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:04.430491 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430387 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tls-certs\") pod \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " Nov 03 00:17:04.430491 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430434 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-dshm\") pod \"3d891d08-d471-4358-9dea-d25276081c2f\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " Nov 03 00:17:04.430491 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430474 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-home\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-home\") pod \"3d891d08-d471-4358-9dea-d25276081c2f\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " Nov 03 00:17:04.431474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430530 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-tmp-dir\") pod \"3d891d08-d471-4358-9dea-d25276081c2f\" (UID: \"3d891d08-d471-4358-9dea-d25276081c2f\") " Nov 03 00:17:04.431474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430560 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tmp-dir\") pod \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\" (UID: \"1fe36396-633b-4974-9fdb-16cf3b7f62c7\") " Nov 03 00:17:04.431474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430953 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.431474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430971 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.431474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.430985 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.431474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.431196 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tmp-dir" pod "1fe36396-633b-4974-9fdb-16cf3b7f62c7" (UID: "1fe36396-633b-4974-9fdb-16cf3b7f62c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:04.431474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.431388 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-home" pod "3d891d08-d471-4358-9dea-d25276081c2f" (UID: "3d891d08-d471-4358-9dea-d25276081c2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:04.431474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.431467 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-tmp-dir" pod "3d891d08-d471-4358-9dea-d25276081c2f" (UID: "3d891d08-d471-4358-9dea-d25276081c2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:04.432737 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.432709 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe36396-633b-4974-9fdb-16cf3b7f62c7-kube-api-access-z6dgj" pod "1fe36396-633b-4974-9fdb-16cf3b7f62c7" (UID: "1fe36396-633b-4974-9fdb-16cf3b7f62c7"). InnerVolumeSpecName "kube-api-access-z6dgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:04.432833 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.432798 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d891d08-d471-4358-9dea-d25276081c2f-tls-certs" pod "3d891d08-d471-4358-9dea-d25276081c2f" (UID: "3d891d08-d471-4358-9dea-d25276081c2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:04.433590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.433563 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tls-certs" pod "1fe36396-633b-4974-9fdb-16cf3b7f62c7" (UID: "1fe36396-633b-4974-9fdb-16cf3b7f62c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:04.433770 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.433749 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d891d08-d471-4358-9dea-d25276081c2f-kube-api-access-q685s" pod "3d891d08-d471-4358-9dea-d25276081c2f" (UID: "3d891d08-d471-4358-9dea-d25276081c2f"). InnerVolumeSpecName "kube-api-access-q685s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:04.434940 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.434904 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-dshm" pod "3d891d08-d471-4358-9dea-d25276081c2f" (UID: "3d891d08-d471-4358-9dea-d25276081c2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:04.436016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.435993 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-dshm" pod "1fe36396-633b-4974-9fdb-16cf3b7f62c7" (UID: "1fe36396-633b-4974-9fdb-16cf3b7f62c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:04.532773 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.532731 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.532773 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.532777 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.533022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.532793 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.533022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.532806 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.533022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.532819 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d891d08-d471-4358-9dea-d25276081c2f-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.533022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.532834 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe36396-633b-4974-9fdb-16cf3b7f62c7-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.533022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.532848 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d891d08-d471-4358-9dea-d25276081c2f-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.533022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.532861 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6dgj\" (UniqueName: \"kubernetes.io/projected/1fe36396-633b-4974-9fdb-16cf3b7f62c7-kube-api-access-z6dgj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.533022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.532875 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q685s\" (UniqueName: \"kubernetes.io/projected/3d891d08-d471-4358-9dea-d25276081c2f-kube-api-access-q685s\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:04.714969 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:04.714886 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44eff2f-b580-4dbe-8ab2-0790894467f6" path="/var/lib/kubelet/pods/d44eff2f-b580-4dbe-8ab2-0790894467f6/volumes" Nov 03 00:17:05.073717 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.073683 5672 generic.go:356] "Generic (PLEG): container finished" podID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerID="cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993" exitCode=0 Nov 03 00:17:05.074178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.073742 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" Nov 03 00:17:05.074178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.073767 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" event={"ID":"1fe36396-633b-4974-9fdb-16cf3b7f62c7","Type":"ContainerDied","Data":"cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993"} Nov 03 00:17:05.074178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.073808 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q" event={"ID":"1fe36396-633b-4974-9fdb-16cf3b7f62c7","Type":"ContainerDied","Data":"84c736a67a5809ba013af01f00ee78a99b3555f56ce2ca34f0323b102077f949"} Nov 03 00:17:05.074178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.073830 5672 scope.go:122] "RemoveContainer" containerID="cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993" Nov 03 00:17:05.075348 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.075320 5672 generic.go:356] "Generic (PLEG): container finished" podID="3d891d08-d471-4358-9dea-d25276081c2f" containerID="3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0" exitCode=0 Nov 03 00:17:05.075475 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.075379 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" Nov 03 00:17:05.075475 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.075411 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" event={"ID":"3d891d08-d471-4358-9dea-d25276081c2f","Type":"ContainerDied","Data":"3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0"} Nov 03 00:17:05.075475 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.075445 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd" event={"ID":"3d891d08-d471-4358-9dea-d25276081c2f","Type":"ContainerDied","Data":"bd7204818f974a0631fe8394ecfb232ab4dce0b44664d44f59b017822a1562c1"} Nov 03 00:17:05.077592 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.077571 5672 generic.go:356] "Generic (PLEG): container finished" podID="a242d460-6053-4ead-8930-bc07066a2e97" containerID="4a571eb3d7bf81ac9bcb5f13051a51a7a0924c763e83c083bffbba9f6beb12b0" exitCode=0 Nov 03 00:17:05.077747 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.077649 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" event={"ID":"a242d460-6053-4ead-8930-bc07066a2e97","Type":"ContainerDied","Data":"4a571eb3d7bf81ac9bcb5f13051a51a7a0924c763e83c083bffbba9f6beb12b0"} Nov 03 00:17:05.083371 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.083355 5672 scope.go:122] "RemoveContainer" containerID="cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993" Nov 03 00:17:05.083623 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:17:05.083605 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993\": container with ID starting with cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993 not found: ID does not exist" containerID="cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993" Nov 03 00:17:05.083668 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.083630 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993"} err="failed to get container status \"cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993\": rpc error: code = NotFound desc = could not find container \"cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993\": container with ID starting with cfc82bbbdfc391b6986ae796f38c97abec064228a9d12457d6e48b5155152993 not found: ID does not exist" Nov 03 00:17:05.083668 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.083646 5672 scope.go:122] "RemoveContainer" containerID="3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0" Nov 03 00:17:05.092118 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.092098 5672 scope.go:122] "RemoveContainer" containerID="3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0" Nov 03 00:17:05.092460 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:17:05.092438 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0\": container with ID starting with 3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0 not found: ID does not exist" containerID="3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0" Nov 03 00:17:05.092520 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.092470 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0"} err="failed to get container status \"3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0\": rpc error: code = NotFound desc = could not find container \"3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0\": container with ID starting with 3cce8c3fc8a32297b48d6c6985e6cc6e95299ce8ec616707d503d8d2a8962fe0 not found: ID does not exist" Nov 03 00:17:05.226581 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.226549 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q"] Nov 03 00:17:05.272646 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.272609 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/premium-w1-3d8404-kserve-75f544c868-fch6q"] Nov 03 00:17:05.436517 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.436431 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd"] Nov 03 00:17:05.439933 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:05.439904 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-2-w1-3d8404-kserve-95dd58556-4tcpd"] Nov 03 00:17:06.225726 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.225704 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:06.352339 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.352252 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-xks-ca\") pod \"a242d460-6053-4ead-8930-bc07066a2e97\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " Nov 03 00:17:06.352536 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.352369 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-openshift-ca\") pod \"a242d460-6053-4ead-8930-bc07066a2e97\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " Nov 03 00:17:06.352536 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.352458 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a242d460-6053-4ead-8930-bc07066a2e97-kube-api-access-7fmnf\" (UniqueName: \"kubernetes.io/projected/a242d460-6053-4ead-8930-bc07066a2e97-kube-api-access-7fmnf\") pod \"a242d460-6053-4ead-8930-bc07066a2e97\" (UID: \"a242d460-6053-4ead-8930-bc07066a2e97\") " Nov 03 00:17:06.353041 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.353009 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-openshift-ca" pod "a242d460-6053-4ead-8930-bc07066a2e97" (UID: "a242d460-6053-4ead-8930-bc07066a2e97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:06.354452 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.354427 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-xks-ca" pod "a242d460-6053-4ead-8930-bc07066a2e97" (UID: "a242d460-6053-4ead-8930-bc07066a2e97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:06.354931 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.354909 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a242d460-6053-4ead-8930-bc07066a2e97-kube-api-access-7fmnf" pod "a242d460-6053-4ead-8930-bc07066a2e97" (UID: "a242d460-6053-4ead-8930-bc07066a2e97"). InnerVolumeSpecName "kube-api-access-7fmnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:06.453315 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.453286 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:06.453315 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.453311 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fmnf\" (UniqueName: \"kubernetes.io/projected/a242d460-6053-4ead-8930-bc07066a2e97-kube-api-access-7fmnf\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:06.453315 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.453322 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a242d460-6053-4ead-8930-bc07066a2e97-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:06.711962 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.711872 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" path="/var/lib/kubelet/pods/1fe36396-633b-4974-9fdb-16cf3b7f62c7/volumes" Nov 03 00:17:06.712329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:06.712314 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d891d08-d471-4358-9dea-d25276081c2f" path="/var/lib/kubelet/pods/3d891d08-d471-4358-9dea-d25276081c2f/volumes" Nov 03 00:17:07.094590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:07.094551 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" event={"ID":"a242d460-6053-4ead-8930-bc07066a2e97","Type":"ContainerDied","Data":"19237f91b95e34ac30b09a7dafbaeed693f39aebb2a4b34f358d0bf452c5ed75"} Nov 03 00:17:07.094774 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:07.094615 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19237f91b95e34ac30b09a7dafbaeed693f39aebb2a4b34f358d0bf452c5ed75" Nov 03 00:17:07.094774 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:07.094561 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt" Nov 03 00:17:17.253331 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.253290 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x"] Nov 03 00:17:17.253984 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.253958 5672 cpu_manager.go:414] "containerMap: removing container" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" Nov 03 00:17:17.254122 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.253988 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" Nov 03 00:17:17.254122 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.254012 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a242d460-6053-4ead-8930-bc07066a2e97" containerName="revoke-keys" Nov 03 00:17:17.254122 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.254022 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a242d460-6053-4ead-8930-bc07066a2e97" containerName="revoke-keys" Nov 03 00:17:17.254122 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.254078 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" Nov 03 00:17:17.254122 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.254087 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" Nov 03 00:17:17.254416 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.254218 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d891d08-d471-4358-9dea-d25276081c2f" containerName="main" Nov 03 00:17:17.254416 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.254233 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe36396-633b-4974-9fdb-16cf3b7f62c7" containerName="main" Nov 03 00:17:17.254416 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.254244 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a242d460-6053-4ead-8930-bc07066a2e97" containerName="revoke-keys" Nov 03 00:17:17.258369 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.258341 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:17.307759 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.307727 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:17:17.312083 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.312055 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x"] Nov 03 00:17:17.354374 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.354339 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:17:17.354607 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.354377 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:17:17.354607 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.354383 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:17:17.362892 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.362866 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjq6\" (UniqueName: \"kubernetes.io/projected/b20c3945-6e41-4ad7-96a6-baf1173a172f-kube-api-access-5pjq6\") pod \"maas-api-revoke-sub-3cefa818-6687x\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:17.363024 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.362928 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cefa818-6687x\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:17.363024 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.362996 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cefa818-6687x\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:17.463652 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.463617 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cefa818-6687x\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:17.463874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.463659 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cefa818-6687x\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:17.463874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.463731 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjq6\" (UniqueName: \"kubernetes.io/projected/b20c3945-6e41-4ad7-96a6-baf1173a172f-kube-api-access-5pjq6\") pod \"maas-api-revoke-sub-3cefa818-6687x\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:17.464537 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.464507 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cefa818-6687x\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:17.466088 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.466064 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cefa818-6687x\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:17.516664 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.516593 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjq6\" (UniqueName: \"kubernetes.io/projected/b20c3945-6e41-4ad7-96a6-baf1173a172f-kube-api-access-5pjq6\") pod \"maas-api-revoke-sub-3cefa818-6687x\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:17.572833 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.572800 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:17.945256 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:17.945224 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x"] Nov 03 00:17:17.945402 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:17:17.945379 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20c3945_6e41_4ad7_96a6_baf1173a172f.slice/crio-f8f9818f03c46928e0d5caf818ac7941512d28a24b8acc3761c42d8833d3cd82 WatchSource:0}: Error finding container f8f9818f03c46928e0d5caf818ac7941512d28a24b8acc3761c42d8833d3cd82: Status 404 returned error can't find the container with id f8f9818f03c46928e0d5caf818ac7941512d28a24b8acc3761c42d8833d3cd82 Nov 03 00:17:18.159783 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:18.159751 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" event={"ID":"b20c3945-6e41-4ad7-96a6-baf1173a172f","Type":"ContainerStarted","Data":"f8f9818f03c46928e0d5caf818ac7941512d28a24b8acc3761c42d8833d3cd82"} Nov 03 00:17:19.169215 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:19.169113 5672 generic.go:356] "Generic (PLEG): container finished" podID="b20c3945-6e41-4ad7-96a6-baf1173a172f" containerID="5db183cb1818e3adba47d6f245a232476850a16cadafb1010eff655ddf6cb0c1" exitCode=0 Nov 03 00:17:19.169215 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:19.169170 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" event={"ID":"b20c3945-6e41-4ad7-96a6-baf1173a172f","Type":"ContainerDied","Data":"5db183cb1818e3adba47d6f245a232476850a16cadafb1010eff655ddf6cb0c1"} Nov 03 00:17:19.992097 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:19.992060 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls"] Nov 03 00:17:19.992352 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:19.992323 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" containerID="cri-o://4edf1262ad260f9ebbb9cafd4db0a12ed607992a6870a215d574bcc5ff74b3f2" gracePeriod=30 Nov 03 00:17:20.048428 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.048369 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s"] Nov 03 00:17:20.051294 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.051268 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5825a40-tnq7s"] Nov 03 00:17:20.312925 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.312901 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:20.392643 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.392610 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-xks-ca\") pod \"b20c3945-6e41-4ad7-96a6-baf1173a172f\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " Nov 03 00:17:20.392643 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.392646 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-openshift-ca\") pod \"b20c3945-6e41-4ad7-96a6-baf1173a172f\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " Nov 03 00:17:20.392864 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.392678 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20c3945-6e41-4ad7-96a6-baf1173a172f-kube-api-access-5pjq6\" (UniqueName: \"kubernetes.io/projected/b20c3945-6e41-4ad7-96a6-baf1173a172f-kube-api-access-5pjq6\") pod \"b20c3945-6e41-4ad7-96a6-baf1173a172f\" (UID: \"b20c3945-6e41-4ad7-96a6-baf1173a172f\") " Nov 03 00:17:20.393043 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.393013 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-openshift-ca" pod "b20c3945-6e41-4ad7-96a6-baf1173a172f" (UID: "b20c3945-6e41-4ad7-96a6-baf1173a172f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:20.394748 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.394716 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-xks-ca" pod "b20c3945-6e41-4ad7-96a6-baf1173a172f" (UID: "b20c3945-6e41-4ad7-96a6-baf1173a172f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:20.394855 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.394834 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20c3945-6e41-4ad7-96a6-baf1173a172f-kube-api-access-5pjq6" pod "b20c3945-6e41-4ad7-96a6-baf1173a172f" (UID: "b20c3945-6e41-4ad7-96a6-baf1173a172f"). InnerVolumeSpecName "kube-api-access-5pjq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:20.494415 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.494387 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:20.494415 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.494413 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20c3945-6e41-4ad7-96a6-baf1173a172f-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:20.494590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.494423 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pjq6\" (UniqueName: \"kubernetes.io/projected/b20c3945-6e41-4ad7-96a6-baf1173a172f-kube-api-access-5pjq6\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:20.713219 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:20.713121 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14742ee8-8770-4a65-98b7-abdb9d054251" path="/var/lib/kubelet/pods/14742ee8-8770-4a65-98b7-abdb9d054251/volumes" Nov 03 00:17:21.182044 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:21.182017 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" Nov 03 00:17:21.182243 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:21.182043 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x" event={"ID":"b20c3945-6e41-4ad7-96a6-baf1173a172f","Type":"ContainerDied","Data":"f8f9818f03c46928e0d5caf818ac7941512d28a24b8acc3761c42d8833d3cd82"} Nov 03 00:17:21.182243 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:21.182084 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8f9818f03c46928e0d5caf818ac7941512d28a24b8acc3761c42d8833d3cd82" Nov 03 00:17:25.208409 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.208377 5672 generic.go:356] "Generic (PLEG): container finished" podID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerID="4edf1262ad260f9ebbb9cafd4db0a12ed607992a6870a215d574bcc5ff74b3f2" exitCode=0 Nov 03 00:17:25.208802 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.208455 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" event={"ID":"9adcfc0e-353b-44e2-b6ee-8a915fac9b52","Type":"ContainerDied","Data":"4edf1262ad260f9ebbb9cafd4db0a12ed607992a6870a215d574bcc5ff74b3f2"} Nov 03 00:17:25.208802 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.208539 5672 scope.go:122] "RemoveContainer" containerID="546303a23c4d2730478e14f596e7ca9561ee2e58eb63df3e605736acb1e2aa24" Nov 03 00:17:25.264768 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.264708 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 03 00:17:25.344818 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.344786 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-kube-api-access-lq87s\" (UniqueName: \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-kube-api-access-lq87s\") pod \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " Nov 03 00:17:25.345016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.344833 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-token\" (UniqueName: \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-token\") pod \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " Nov 03 00:17:25.345016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.344866 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-data\") pod \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " Nov 03 00:17:25.345016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.344904 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istiod-ca-cert\") pod \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " Nov 03 00:17:25.345016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.344949 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-socket\") pod \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " Nov 03 00:17:25.345016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.344993 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-envoy\") pod \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " Nov 03 00:17:25.345324 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.345056 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-podinfo\") pod \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " Nov 03 00:17:25.345324 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.345134 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-credential-socket\") pod \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " Nov 03 00:17:25.345324 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.345217 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-certs\") pod \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\" (UID: \"9adcfc0e-353b-44e2-b6ee-8a915fac9b52\") " Nov 03 00:17:25.345491 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.345434 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istiod-ca-cert" pod "9adcfc0e-353b-44e2-b6ee-8a915fac9b52" (UID: "9adcfc0e-353b-44e2-b6ee-8a915fac9b52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:25.345491 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.345471 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-socket" pod "9adcfc0e-353b-44e2-b6ee-8a915fac9b52" (UID: "9adcfc0e-353b-44e2-b6ee-8a915fac9b52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.345639 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.345620 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.345704 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.345644 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.345704 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.345650 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-certs" pod "9adcfc0e-353b-44e2-b6ee-8a915fac9b52" (UID: "9adcfc0e-353b-44e2-b6ee-8a915fac9b52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.345933 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.345844 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-credential-socket" pod "9adcfc0e-353b-44e2-b6ee-8a915fac9b52" (UID: "9adcfc0e-353b-44e2-b6ee-8a915fac9b52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.345933 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.345909 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-data" pod "9adcfc0e-353b-44e2-b6ee-8a915fac9b52" (UID: "9adcfc0e-353b-44e2-b6ee-8a915fac9b52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.347688 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.347666 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-envoy" pod "9adcfc0e-353b-44e2-b6ee-8a915fac9b52" (UID: "9adcfc0e-353b-44e2-b6ee-8a915fac9b52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:25.348241 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.348134 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-token" pod "9adcfc0e-353b-44e2-b6ee-8a915fac9b52" (UID: "9adcfc0e-353b-44e2-b6ee-8a915fac9b52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:25.348241 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.348173 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-kube-api-access-lq87s" pod "9adcfc0e-353b-44e2-b6ee-8a915fac9b52" (UID: "9adcfc0e-353b-44e2-b6ee-8a915fac9b52"). InnerVolumeSpecName "kube-api-access-lq87s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:25.349133 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.349077 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-podinfo" pod "9adcfc0e-353b-44e2-b6ee-8a915fac9b52" (UID: "9adcfc0e-353b-44e2-b6ee-8a915fac9b52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:17:25.446564 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.446523 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.446564 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.446558 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.446564 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.446571 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.446820 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.446583 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.446820 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.446598 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq87s\" (UniqueName: \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-kube-api-access-lq87s\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.446820 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.446610 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:25.446820 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:25.446623 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9adcfc0e-353b-44e2-b6ee-8a915fac9b52-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:26.019623 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:17:26.019586 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:17:26.215256 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:26.215223 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" Nov 03 00:17:26.215256 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:26.215247 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls" event={"ID":"9adcfc0e-353b-44e2-b6ee-8a915fac9b52","Type":"ContainerDied","Data":"0aa6fcf77e69c57206220f5b8d5095193b1693944a514f0c2bbd765cbbb1832b"} Nov 03 00:17:26.215779 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:26.215319 5672 scope.go:122] "RemoveContainer" containerID="4edf1262ad260f9ebbb9cafd4db0a12ed607992a6870a215d574bcc5ff74b3f2" Nov 03 00:17:26.415632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:26.415552 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls"] Nov 03 00:17:26.422821 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:26.422793 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-28f82c-openshift-default-86bbcfbf95-x5pls"] Nov 03 00:17:26.712791 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:26.712720 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" path="/var/lib/kubelet/pods/9adcfc0e-353b-44e2-b6ee-8a915fac9b52/volumes" Nov 03 00:17:27.219336 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.219301 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh"] Nov 03 00:17:27.924381 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.924345 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z"] Nov 03 00:17:27.924861 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.924840 5672 cpu_manager.go:414] "containerMap: removing container" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" Nov 03 00:17:27.924953 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.924863 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" Nov 03 00:17:27.924953 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.924912 5672 cpu_manager.go:414] "containerMap: removing container" podUID="b20c3945-6e41-4ad7-96a6-baf1173a172f" containerName="revoke-keys" Nov 03 00:17:27.924953 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.924921 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20c3945-6e41-4ad7-96a6-baf1173a172f" containerName="revoke-keys" Nov 03 00:17:27.925117 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.925015 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20c3945-6e41-4ad7-96a6-baf1173a172f" containerName="revoke-keys" Nov 03 00:17:27.925117 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.925035 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" Nov 03 00:17:27.925117 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.925049 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" Nov 03 00:17:27.931302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.931276 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:27.980457 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.980343 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:17:27.980610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.980532 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:17:27.980610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.980585 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:17:27.980742 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.980612 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:17:27.982011 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:27.981943 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z"] Nov 03 00:17:28.075410 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:28.075368 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v46h\" (UniqueName: \"kubernetes.io/projected/b8991896-637e-4812-a82b-d2e9175f1567-kube-api-access-6v46h\") pod \"maas-api-revoke-sub-c35399aa-lj57z\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:28.075585 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:28.075555 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35399aa-lj57z\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:28.075647 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:28.075585 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35399aa-lj57z\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:28.176117 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:28.176034 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35399aa-lj57z\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:28.176117 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:28.176069 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35399aa-lj57z\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:28.176384 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:28.176184 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v46h\" (UniqueName: \"kubernetes.io/projected/b8991896-637e-4812-a82b-d2e9175f1567-kube-api-access-6v46h\") pod \"maas-api-revoke-sub-c35399aa-lj57z\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:28.176842 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:28.176815 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c35399aa-lj57z\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:28.178638 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:28.178614 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c35399aa-lj57z\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:28.231083 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:28.231057 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v46h\" (UniqueName: \"kubernetes.io/projected/b8991896-637e-4812-a82b-d2e9175f1567-kube-api-access-6v46h\") pod \"maas-api-revoke-sub-c35399aa-lj57z\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:28.249681 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:28.249658 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:28.627051 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:28.627015 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z"] Nov 03 00:17:28.635851 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:17:28.635732 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8991896_637e_4812_a82b_d2e9175f1567.slice/crio-ccdb8f32f946874ae9c3a3e58b855c1b39abf0aa853feabf91bcbc44ec7e28b3 WatchSource:0}: Error finding container ccdb8f32f946874ae9c3a3e58b855c1b39abf0aa853feabf91bcbc44ec7e28b3: Status 404 returned error can't find the container with id ccdb8f32f946874ae9c3a3e58b855c1b39abf0aa853feabf91bcbc44ec7e28b3 Nov 03 00:17:29.243999 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:29.243962 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" event={"ID":"b8991896-637e-4812-a82b-d2e9175f1567","Type":"ContainerStarted","Data":"7a19f8dd5d154aa1605c1c7275c4334d76215dd0358ec538e63f0b149875cb4f"} Nov 03 00:17:29.244521 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:29.244021 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" event={"ID":"b8991896-637e-4812-a82b-d2e9175f1567","Type":"ContainerStarted","Data":"ccdb8f32f946874ae9c3a3e58b855c1b39abf0aa853feabf91bcbc44ec7e28b3"} Nov 03 00:17:29.351555 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:29.351486 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" podStartSLOduration=2.351468399 podStartE2EDuration="2.351468399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:29.349312509 +0000 UTC m=+5042.756624354" watchObservedRunningTime="2026-11-03 00:17:29.351468399 +0000 UTC m=+5042.758780231" Nov 03 00:17:30.251473 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:30.251380 5672 generic.go:356] "Generic (PLEG): container finished" podID="b8991896-637e-4812-a82b-d2e9175f1567" containerID="7a19f8dd5d154aa1605c1c7275c4334d76215dd0358ec538e63f0b149875cb4f" exitCode=0 Nov 03 00:17:30.251473 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:30.251439 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" event={"ID":"b8991896-637e-4812-a82b-d2e9175f1567","Type":"ContainerDied","Data":"7a19f8dd5d154aa1605c1c7275c4334d76215dd0358ec538e63f0b149875cb4f"} Nov 03 00:17:31.397162 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:31.397124 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:31.506245 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:31.506214 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-xks-ca\") pod \"b8991896-637e-4812-a82b-d2e9175f1567\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " Nov 03 00:17:31.506445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:31.506258 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8991896-637e-4812-a82b-d2e9175f1567-kube-api-access-6v46h\" (UniqueName: \"kubernetes.io/projected/b8991896-637e-4812-a82b-d2e9175f1567-kube-api-access-6v46h\") pod \"b8991896-637e-4812-a82b-d2e9175f1567\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " Nov 03 00:17:31.506445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:31.506401 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-openshift-ca\") pod \"b8991896-637e-4812-a82b-d2e9175f1567\" (UID: \"b8991896-637e-4812-a82b-d2e9175f1567\") " Nov 03 00:17:31.506753 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:31.506724 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-openshift-ca" pod "b8991896-637e-4812-a82b-d2e9175f1567" (UID: "b8991896-637e-4812-a82b-d2e9175f1567"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:31.508454 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:31.508435 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8991896-637e-4812-a82b-d2e9175f1567-kube-api-access-6v46h" pod "b8991896-637e-4812-a82b-d2e9175f1567" (UID: "b8991896-637e-4812-a82b-d2e9175f1567"). InnerVolumeSpecName "kube-api-access-6v46h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:31.508538 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:31.508518 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-xks-ca" pod "b8991896-637e-4812-a82b-d2e9175f1567" (UID: "b8991896-637e-4812-a82b-d2e9175f1567"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:31.607544 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:31.607511 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:31.607544 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:31.607540 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8991896-637e-4812-a82b-d2e9175f1567-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:31.607544 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:31.607551 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v46h\" (UniqueName: \"kubernetes.io/projected/b8991896-637e-4812-a82b-d2e9175f1567-kube-api-access-6v46h\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:32.220558 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:32.220515 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" podUID="a95cee42-e848-4f4b-98e5-c4c9c2c00c3a" containerName="payload-processing" containerID="cri-o://47da314c6eaf4bd79da7b66c959f7d2ba0ef8ac01682281095c2427036ab7352" gracePeriod=25 Nov 03 00:17:32.267864 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:32.267832 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" Nov 03 00:17:32.268039 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:32.267863 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z" event={"ID":"b8991896-637e-4812-a82b-d2e9175f1567","Type":"ContainerDied","Data":"ccdb8f32f946874ae9c3a3e58b855c1b39abf0aa853feabf91bcbc44ec7e28b3"} Nov 03 00:17:32.268039 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:32.267923 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccdb8f32f946874ae9c3a3e58b855c1b39abf0aa853feabf91bcbc44ec7e28b3" Nov 03 00:17:39.132740 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.132534 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr"] Nov 03 00:17:39.133276 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.132791 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" podUID="96526185-0fff-403d-83df-146ecdbd24e0" containerName="istio-proxy" containerID="cri-o://6021d421cf0aa3fbb993abf198674d231cc549a8bac2f6effb822f95ca397ae5" gracePeriod=30 Nov 03 00:17:39.581741 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.581709 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg"] Nov 03 00:17:39.582174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.582160 5672 cpu_manager.go:414] "containerMap: removing container" podUID="b8991896-637e-4812-a82b-d2e9175f1567" containerName="revoke-keys" Nov 03 00:17:39.582227 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.582176 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8991896-637e-4812-a82b-d2e9175f1567" containerName="revoke-keys" Nov 03 00:17:39.582227 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.582210 5672 cpu_manager.go:414] "containerMap: removing container" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" Nov 03 00:17:39.582227 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.582215 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9adcfc0e-353b-44e2-b6ee-8a915fac9b52" containerName="istio-proxy" Nov 03 00:17:39.582357 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.582274 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8991896-637e-4812-a82b-d2e9175f1567" containerName="revoke-keys" Nov 03 00:17:39.585860 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.585841 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:39.633959 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.633931 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:17:39.634262 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.634233 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:17:39.634527 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.634509 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:17:39.634615 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.634529 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:17:39.641493 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.641464 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg"] Nov 03 00:17:39.689067 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.689033 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f2cj\" (UniqueName: \"kubernetes.io/projected/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-kube-api-access-8f2cj\") pod \"maas-api-revoke-sub-10f92101-vgnsg\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:39.689260 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.689079 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f92101-vgnsg\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:39.689260 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.689238 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f92101-vgnsg\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:39.790374 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.790339 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f92101-vgnsg\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:39.790555 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.790412 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2cj\" (UniqueName: \"kubernetes.io/projected/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-kube-api-access-8f2cj\") pod \"maas-api-revoke-sub-10f92101-vgnsg\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:39.790555 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.790438 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f92101-vgnsg\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:39.791041 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.791014 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f92101-vgnsg\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:39.792979 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.792954 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f92101-vgnsg\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:39.843948 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.843866 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2cj\" (UniqueName: \"kubernetes.io/projected/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-kube-api-access-8f2cj\") pod \"maas-api-revoke-sub-10f92101-vgnsg\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:39.900703 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:39.900670 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:40.266832 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:40.266801 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg"] Nov 03 00:17:40.270326 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:17:40.270283 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7ba1f4_fbe0_4c8e_b334_3959010b1fb9.slice/crio-3c2755443e7668833b95eeaccf42d78c88d3206b7ddbadb19dd025e01aa4ca7f WatchSource:0}: Error finding container 3c2755443e7668833b95eeaccf42d78c88d3206b7ddbadb19dd025e01aa4ca7f: Status 404 returned error can't find the container with id 3c2755443e7668833b95eeaccf42d78c88d3206b7ddbadb19dd025e01aa4ca7f Nov 03 00:17:40.316256 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:40.316221 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" event={"ID":"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9","Type":"ContainerStarted","Data":"3c2755443e7668833b95eeaccf42d78c88d3206b7ddbadb19dd025e01aa4ca7f"} Nov 03 00:17:41.326315 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:41.326271 5672 generic.go:356] "Generic (PLEG): container finished" podID="de7ba1f4-fbe0-4c8e-b334-3959010b1fb9" containerID="6412016d944250b619db0ca7b7c9ba25210b57c6e7ddde028fb1174e92d90288" exitCode=0 Nov 03 00:17:41.326850 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:41.326362 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" event={"ID":"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9","Type":"ContainerDied","Data":"6412016d944250b619db0ca7b7c9ba25210b57c6e7ddde028fb1174e92d90288"} Nov 03 00:17:42.101261 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:42.101228 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" podUID="a95cee42-e848-4f4b-98e5-c4c9c2c00c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:17:42.486088 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:42.486061 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:42.616633 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:42.616600 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-openshift-ca\") pod \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " Nov 03 00:17:42.616832 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:42.616734 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-kube-api-access-8f2cj\" (UniqueName: \"kubernetes.io/projected/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-kube-api-access-8f2cj\") pod \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " Nov 03 00:17:42.616832 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:42.616785 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-xks-ca\") pod \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\" (UID: \"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9\") " Nov 03 00:17:42.617136 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:42.617001 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-openshift-ca" pod "de7ba1f4-fbe0-4c8e-b334-3959010b1fb9" (UID: "de7ba1f4-fbe0-4c8e-b334-3959010b1fb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:42.617443 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:42.617421 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:42.618864 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:42.618842 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-kube-api-access-8f2cj" pod "de7ba1f4-fbe0-4c8e-b334-3959010b1fb9" (UID: "de7ba1f4-fbe0-4c8e-b334-3959010b1fb9"). InnerVolumeSpecName "kube-api-access-8f2cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:42.618987 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:42.618970 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-xks-ca" pod "de7ba1f4-fbe0-4c8e-b334-3959010b1fb9" (UID: "de7ba1f4-fbe0-4c8e-b334-3959010b1fb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:42.718738 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:42.718709 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f2cj\" (UniqueName: \"kubernetes.io/projected/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-kube-api-access-8f2cj\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:42.718738 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:42.718736 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:43.339844 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:43.339816 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" Nov 03 00:17:43.340021 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:43.339836 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg" event={"ID":"de7ba1f4-fbe0-4c8e-b334-3959010b1fb9","Type":"ContainerDied","Data":"3c2755443e7668833b95eeaccf42d78c88d3206b7ddbadb19dd025e01aa4ca7f"} Nov 03 00:17:43.340021 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:43.339894 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c2755443e7668833b95eeaccf42d78c88d3206b7ddbadb19dd025e01aa4ca7f" Nov 03 00:17:43.912810 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:43.912769 5672 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:17:43.913258 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:43.912842 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" podUID="96526185-0fff-403d-83df-146ecdbd24e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:17:44.349992 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.349954 5672 generic.go:356] "Generic (PLEG): container finished" podID="96526185-0fff-403d-83df-146ecdbd24e0" containerID="6021d421cf0aa3fbb993abf198674d231cc549a8bac2f6effb822f95ca397ae5" exitCode=0 Nov 03 00:17:44.350185 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.350009 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" event={"ID":"96526185-0fff-403d-83df-146ecdbd24e0","Type":"ContainerDied","Data":"6021d421cf0aa3fbb993abf198674d231cc549a8bac2f6effb822f95ca397ae5"} Nov 03 00:17:44.388908 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.388882 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 03 00:17:44.536611 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.536529 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-credential-socket\") pod \"96526185-0fff-403d-83df-146ecdbd24e0\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " Nov 03 00:17:44.536611 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.536573 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-envoy\") pod \"96526185-0fff-403d-83df-146ecdbd24e0\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " Nov 03 00:17:44.536611 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.536611 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-data\") pod \"96526185-0fff-403d-83df-146ecdbd24e0\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " Nov 03 00:17:44.536921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.536676 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-kube-api-access-5ccbm\" (UniqueName: \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-kube-api-access-5ccbm\") pod \"96526185-0fff-403d-83df-146ecdbd24e0\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " Nov 03 00:17:44.536921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.536701 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-socket\") pod \"96526185-0fff-403d-83df-146ecdbd24e0\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " Nov 03 00:17:44.536921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.536750 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-istio-token\" (UniqueName: \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-istio-token\") pod \"96526185-0fff-403d-83df-146ecdbd24e0\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " Nov 03 00:17:44.536921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.536778 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-certs\") pod \"96526185-0fff-403d-83df-146ecdbd24e0\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " Nov 03 00:17:44.536921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.536815 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96526185-0fff-403d-83df-146ecdbd24e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96526185-0fff-403d-83df-146ecdbd24e0-istio-podinfo\") pod \"96526185-0fff-403d-83df-146ecdbd24e0\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " Nov 03 00:17:44.536921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.536862 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96526185-0fff-403d-83df-146ecdbd24e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96526185-0fff-403d-83df-146ecdbd24e0-istiod-ca-cert\") pod \"96526185-0fff-403d-83df-146ecdbd24e0\" (UID: \"96526185-0fff-403d-83df-146ecdbd24e0\") " Nov 03 00:17:44.536921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.536863 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-credential-socket" pod "96526185-0fff-403d-83df-146ecdbd24e0" (UID: "96526185-0fff-403d-83df-146ecdbd24e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:44.537337 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.536976 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-data" pod "96526185-0fff-403d-83df-146ecdbd24e0" (UID: "96526185-0fff-403d-83df-146ecdbd24e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:44.537337 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.537110 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-socket" pod "96526185-0fff-403d-83df-146ecdbd24e0" (UID: "96526185-0fff-403d-83df-146ecdbd24e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:44.537337 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.537133 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-certs" pod "96526185-0fff-403d-83df-146ecdbd24e0" (UID: "96526185-0fff-403d-83df-146ecdbd24e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:44.537499 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.537338 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.537499 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.537358 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.537499 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.537360 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96526185-0fff-403d-83df-146ecdbd24e0-istiod-ca-cert" pod "96526185-0fff-403d-83df-146ecdbd24e0" (UID: "96526185-0fff-403d-83df-146ecdbd24e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:44.537499 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.537374 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.537499 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.537389 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.539042 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.539017 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-kube-api-access-5ccbm" pod "96526185-0fff-403d-83df-146ecdbd24e0" (UID: "96526185-0fff-403d-83df-146ecdbd24e0"). InnerVolumeSpecName "kube-api-access-5ccbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:44.539237 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.539136 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96526185-0fff-403d-83df-146ecdbd24e0-istio-podinfo" pod "96526185-0fff-403d-83df-146ecdbd24e0" (UID: "96526185-0fff-403d-83df-146ecdbd24e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:17:44.539305 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.539280 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-envoy" pod "96526185-0fff-403d-83df-146ecdbd24e0" (UID: "96526185-0fff-403d-83df-146ecdbd24e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:17:44.539305 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.539296 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-istio-token" pod "96526185-0fff-403d-83df-146ecdbd24e0" (UID: "96526185-0fff-403d-83df-146ecdbd24e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:44.638638 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.638601 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ccbm\" (UniqueName: \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-kube-api-access-5ccbm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.638638 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.638636 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96526185-0fff-403d-83df-146ecdbd24e0-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.638638 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.638646 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96526185-0fff-403d-83df-146ecdbd24e0-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.638870 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.638655 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96526185-0fff-403d-83df-146ecdbd24e0-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:44.638870 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:44.638664 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96526185-0fff-403d-83df-146ecdbd24e0-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:45.356827 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:45.356788 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" event={"ID":"96526185-0fff-403d-83df-146ecdbd24e0","Type":"ContainerDied","Data":"68f270e966e148dca99f170243996acad3657be50be1a618acb1e34e0f7e42ac"} Nov 03 00:17:45.357341 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:45.356840 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr" Nov 03 00:17:45.357341 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:45.356862 5672 scope.go:122] "RemoveContainer" containerID="6021d421cf0aa3fbb993abf198674d231cc549a8bac2f6effb822f95ca397ae5" Nov 03 00:17:45.530278 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:45.530203 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr"] Nov 03 00:17:45.577592 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:45.577536 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-tq8kr"] Nov 03 00:17:46.711853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:46.711821 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96526185-0fff-403d-83df-146ecdbd24e0" path="/var/lib/kubelet/pods/96526185-0fff-403d-83df-146ecdbd24e0/volumes" Nov 03 00:17:50.573174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.573108 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r"] Nov 03 00:17:50.573755 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.573732 5672 cpu_manager.go:414] "containerMap: removing container" podUID="de7ba1f4-fbe0-4c8e-b334-3959010b1fb9" containerName="revoke-keys" Nov 03 00:17:50.573847 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.573757 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7ba1f4-fbe0-4c8e-b334-3959010b1fb9" containerName="revoke-keys" Nov 03 00:17:50.573847 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.573782 5672 cpu_manager.go:414] "containerMap: removing container" podUID="96526185-0fff-403d-83df-146ecdbd24e0" containerName="istio-proxy" Nov 03 00:17:50.573847 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.573790 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96526185-0fff-403d-83df-146ecdbd24e0" containerName="istio-proxy" Nov 03 00:17:50.574005 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.573922 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7ba1f4-fbe0-4c8e-b334-3959010b1fb9" containerName="revoke-keys" Nov 03 00:17:50.574005 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.573937 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="96526185-0fff-403d-83df-146ecdbd24e0" containerName="istio-proxy" Nov 03 00:17:50.578648 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.578625 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:50.584351 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.584327 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f5b73a-fvv2r\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:50.584462 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.584448 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f5b73a-fvv2r\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:50.584512 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.584491 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42d8p\" (UniqueName: \"kubernetes.io/projected/b39c9d5b-887e-4ec0-ae61-011842d2af4c-kube-api-access-42d8p\") pod \"maas-api-revoke-sub-89f5b73a-fvv2r\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:50.632718 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.632671 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r"] Nov 03 00:17:50.675632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.675588 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:17:50.675632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.675597 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:17:50.675632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.675635 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:17:50.675885 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.675750 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:17:50.685075 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.685051 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f5b73a-fvv2r\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:50.685212 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.685124 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f5b73a-fvv2r\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:50.685212 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.685197 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42d8p\" (UniqueName: \"kubernetes.io/projected/b39c9d5b-887e-4ec0-ae61-011842d2af4c-kube-api-access-42d8p\") pod \"maas-api-revoke-sub-89f5b73a-fvv2r\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:50.685733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.685710 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f5b73a-fvv2r\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:50.687440 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.687421 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f5b73a-fvv2r\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:50.739438 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.739403 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42d8p\" (UniqueName: \"kubernetes.io/projected/b39c9d5b-887e-4ec0-ae61-011842d2af4c-kube-api-access-42d8p\") pod \"maas-api-revoke-sub-89f5b73a-fvv2r\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:50.891096 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:50.891015 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:51.275643 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:17:51.275601 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39c9d5b_887e_4ec0_ae61_011842d2af4c.slice/crio-bbbb258cb3b70d37fb2ce149efafb957145e36d3868b54e199bfc1d7bf1f600e WatchSource:0}: Error finding container bbbb258cb3b70d37fb2ce149efafb957145e36d3868b54e199bfc1d7bf1f600e: Status 404 returned error can't find the container with id bbbb258cb3b70d37fb2ce149efafb957145e36d3868b54e199bfc1d7bf1f600e Nov 03 00:17:51.284686 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:51.284659 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r"] Nov 03 00:17:51.392830 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:51.392800 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" event={"ID":"b39c9d5b-887e-4ec0-ae61-011842d2af4c","Type":"ContainerStarted","Data":"bbbb258cb3b70d37fb2ce149efafb957145e36d3868b54e199bfc1d7bf1f600e"} Nov 03 00:17:52.099693 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.099659 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" podUID="a95cee42-e848-4f4b-98e5-c4c9c2c00c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:17:52.404442 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.404403 5672 generic.go:356] "Generic (PLEG): container finished" podID="a95cee42-e848-4f4b-98e5-c4c9c2c00c3a" containerID="47da314c6eaf4bd79da7b66c959f7d2ba0ef8ac01682281095c2427036ab7352" exitCode=0 Nov 03 00:17:52.404608 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.404434 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" event={"ID":"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a","Type":"ContainerDied","Data":"47da314c6eaf4bd79da7b66c959f7d2ba0ef8ac01682281095c2427036ab7352"} Nov 03 00:17:52.406392 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.406369 5672 generic.go:356] "Generic (PLEG): container finished" podID="b39c9d5b-887e-4ec0-ae61-011842d2af4c" containerID="f2b526e2d3eafabec2e0029ac1377ff531adbb67657ae23f2134f56e6defc1d6" exitCode=0 Nov 03 00:17:52.406519 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.406446 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" event={"ID":"b39c9d5b-887e-4ec0-ae61-011842d2af4c","Type":"ContainerDied","Data":"f2b526e2d3eafabec2e0029ac1377ff531adbb67657ae23f2134f56e6defc1d6"} Nov 03 00:17:52.488277 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.488251 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 03 00:17:52.499466 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.499439 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-plugins-config-volume\") pod \"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a\" (UID: \"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a\") " Nov 03 00:17:52.499776 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.499751 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-plugins-config-volume" pod "a95cee42-e848-4f4b-98e5-c4c9c2c00c3a" (UID: "a95cee42-e848-4f4b-98e5-c4c9c2c00c3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:52.600437 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.600358 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-kube-api-access-zm45q\" (UniqueName: \"kubernetes.io/projected/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-kube-api-access-zm45q\") pod \"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a\" (UID: \"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a\") " Nov 03 00:17:52.600682 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.600662 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:52.602471 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.602442 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-kube-api-access-zm45q" pod "a95cee42-e848-4f4b-98e5-c4c9c2c00c3a" (UID: "a95cee42-e848-4f4b-98e5-c4c9c2c00c3a"). InnerVolumeSpecName "kube-api-access-zm45q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:52.701449 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:52.701410 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm45q\" (UniqueName: \"kubernetes.io/projected/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a-kube-api-access-zm45q\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:53.414349 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.414315 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" Nov 03 00:17:53.414349 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.414334 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh" event={"ID":"a95cee42-e848-4f4b-98e5-c4c9c2c00c3a","Type":"ContainerDied","Data":"5c88b8e09f4e537827cc48258bd3327c7a4ebae318abaed06b5e30a32bc51261"} Nov 03 00:17:53.414876 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.414403 5672 scope.go:122] "RemoveContainer" containerID="47da314c6eaf4bd79da7b66c959f7d2ba0ef8ac01682281095c2427036ab7352" Nov 03 00:17:53.551896 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.551868 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:53.580893 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.580858 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh"] Nov 03 00:17:53.590317 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.590285 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7ca60f-b5fd89695-94bdh"] Nov 03 00:17:53.609736 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.609705 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-openshift-ca\") pod \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " Nov 03 00:17:53.609904 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.609788 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-xks-ca\") pod \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " Nov 03 00:17:53.609904 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.609829 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39c9d5b-887e-4ec0-ae61-011842d2af4c-kube-api-access-42d8p\" (UniqueName: \"kubernetes.io/projected/b39c9d5b-887e-4ec0-ae61-011842d2af4c-kube-api-access-42d8p\") pod \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\" (UID: \"b39c9d5b-887e-4ec0-ae61-011842d2af4c\") " Nov 03 00:17:53.610105 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.610080 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-openshift-ca" pod "b39c9d5b-887e-4ec0-ae61-011842d2af4c" (UID: "b39c9d5b-887e-4ec0-ae61-011842d2af4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:17:53.611928 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.611902 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-xks-ca" pod "b39c9d5b-887e-4ec0-ae61-011842d2af4c" (UID: "b39c9d5b-887e-4ec0-ae61-011842d2af4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:17:53.612032 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.611903 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39c9d5b-887e-4ec0-ae61-011842d2af4c-kube-api-access-42d8p" pod "b39c9d5b-887e-4ec0-ae61-011842d2af4c" (UID: "b39c9d5b-887e-4ec0-ae61-011842d2af4c"). InnerVolumeSpecName "kube-api-access-42d8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:17:53.711304 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.711052 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:53.711304 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.711087 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39c9d5b-887e-4ec0-ae61-011842d2af4c-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:53.711304 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:53.711103 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42d8p\" (UniqueName: \"kubernetes.io/projected/b39c9d5b-887e-4ec0-ae61-011842d2af4c-kube-api-access-42d8p\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:17:54.423410 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:54.423382 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" Nov 03 00:17:54.423876 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:54.423408 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r" event={"ID":"b39c9d5b-887e-4ec0-ae61-011842d2af4c","Type":"ContainerDied","Data":"bbbb258cb3b70d37fb2ce149efafb957145e36d3868b54e199bfc1d7bf1f600e"} Nov 03 00:17:54.423876 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:54.423468 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbbb258cb3b70d37fb2ce149efafb957145e36d3868b54e199bfc1d7bf1f600e" Nov 03 00:17:54.711243 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:17:54.711171 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a95cee42-e848-4f4b-98e5-c4c9c2c00c3a" path="/var/lib/kubelet/pods/a95cee42-e848-4f4b-98e5-c4c9c2c00c3a/volumes" Nov 03 00:17:56.020703 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:17:56.020668 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:18:04.066545 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:04.066509 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr"] Nov 03 00:18:04.071197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:04.071171 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:18:04.112627 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:04.112598 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b0cc116-b6jfr"] Nov 03 00:18:04.250733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:04.250697 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg"] Nov 03 00:18:04.250948 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:04.250908 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" containerID="cri-o://92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009" gracePeriod=30 Nov 03 00:18:04.712125 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:04.712090 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e28074d5-66da-451a-9bc0-fd947c8c9942" path="/var/lib/kubelet/pods/e28074d5-66da-451a-9bc0-fd947c8c9942/volumes" Nov 03 00:18:05.336168 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.336118 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd"] Nov 03 00:18:05.336568 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.336554 5672 cpu_manager.go:414] "containerMap: removing container" podUID="b39c9d5b-887e-4ec0-ae61-011842d2af4c" containerName="revoke-keys" Nov 03 00:18:05.336622 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.336569 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39c9d5b-887e-4ec0-ae61-011842d2af4c" containerName="revoke-keys" Nov 03 00:18:05.336622 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.336586 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a95cee42-e848-4f4b-98e5-c4c9c2c00c3a" containerName="payload-processing" Nov 03 00:18:05.336622 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.336591 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95cee42-e848-4f4b-98e5-c4c9c2c00c3a" containerName="payload-processing" Nov 03 00:18:05.336721 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.336663 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39c9d5b-887e-4ec0-ae61-011842d2af4c" containerName="revoke-keys" Nov 03 00:18:05.336721 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.336674 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a95cee42-e848-4f4b-98e5-c4c9c2c00c3a" containerName="payload-processing" Nov 03 00:18:05.341374 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.341355 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:05.396752 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.396725 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd"] Nov 03 00:18:05.421780 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.421753 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6gjt\" (UniqueName: \"kubernetes.io/projected/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-kube-api-access-s6gjt\") pod \"maas-api-revoke-sub-b99f726d-tkjkd\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:05.421936 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.421882 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b99f726d-tkjkd\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:05.422000 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.421942 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b99f726d-tkjkd\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:05.439833 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.439806 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:18:05.439973 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.439831 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:18:05.439973 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.439838 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:18:05.439973 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.439871 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:18:05.522562 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.522535 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b99f726d-tkjkd\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:05.522702 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.522598 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6gjt\" (UniqueName: \"kubernetes.io/projected/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-kube-api-access-s6gjt\") pod \"maas-api-revoke-sub-b99f726d-tkjkd\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:05.522702 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.522690 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b99f726d-tkjkd\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:05.523276 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.523257 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b99f726d-tkjkd\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:05.524933 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.524914 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b99f726d-tkjkd\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:05.578021 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.577991 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6gjt\" (UniqueName: \"kubernetes.io/projected/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-kube-api-access-s6gjt\") pod \"maas-api-revoke-sub-b99f726d-tkjkd\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:05.654244 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:05.654172 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:06.027408 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:06.027374 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd"] Nov 03 00:18:06.028780 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:18:06.028753 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf456c7bf_4939_4c59_afd0_d7e8ad0762e8.slice/crio-f6734a5d7738c9727fdbd694ad04b32ec7b269ff8572affb3a19c523c7607f50 WatchSource:0}: Error finding container f6734a5d7738c9727fdbd694ad04b32ec7b269ff8572affb3a19c523c7607f50: Status 404 returned error can't find the container with id f6734a5d7738c9727fdbd694ad04b32ec7b269ff8572affb3a19c523c7607f50 Nov 03 00:18:06.512959 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:06.512926 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" event={"ID":"f456c7bf-4939-4c59-afd0-d7e8ad0762e8","Type":"ContainerStarted","Data":"d0d35f226d620b256b2d07c73ed989d6af99068c1966bfa9180c4138e2e9d040"} Nov 03 00:18:06.513355 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:06.512981 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" event={"ID":"f456c7bf-4939-4c59-afd0-d7e8ad0762e8","Type":"ContainerStarted","Data":"f6734a5d7738c9727fdbd694ad04b32ec7b269ff8572affb3a19c523c7607f50"} Nov 03 00:18:06.623080 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:06.622966 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" podStartSLOduration=1.6229486149999999 podStartE2EDuration="1.622948615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:06.62247021 +0000 UTC m=+5080.029782043" watchObservedRunningTime="2026-11-03 00:18:06.622948615 +0000 UTC m=+5080.030260446" Nov 03 00:18:07.519933 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:07.519899 5672 generic.go:356] "Generic (PLEG): container finished" podID="f456c7bf-4939-4c59-afd0-d7e8ad0762e8" containerID="d0d35f226d620b256b2d07c73ed989d6af99068c1966bfa9180c4138e2e9d040" exitCode=0 Nov 03 00:18:07.520385 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:07.519990 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" event={"ID":"f456c7bf-4939-4c59-afd0-d7e8ad0762e8","Type":"ContainerDied","Data":"d0d35f226d620b256b2d07c73ed989d6af99068c1966bfa9180c4138e2e9d040"} Nov 03 00:18:08.662466 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:08.662442 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:08.752448 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:08.752419 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-kube-api-access-s6gjt\" (UniqueName: \"kubernetes.io/projected/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-kube-api-access-s6gjt\") pod \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " Nov 03 00:18:08.752619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:08.752480 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-openshift-ca\") pod \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " Nov 03 00:18:08.752619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:08.752547 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-xks-ca\") pod \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\" (UID: \"f456c7bf-4939-4c59-afd0-d7e8ad0762e8\") " Nov 03 00:18:08.752852 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:08.752827 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-openshift-ca" pod "f456c7bf-4939-4c59-afd0-d7e8ad0762e8" (UID: "f456c7bf-4939-4c59-afd0-d7e8ad0762e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:18:08.754639 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:08.754613 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-xks-ca" pod "f456c7bf-4939-4c59-afd0-d7e8ad0762e8" (UID: "f456c7bf-4939-4c59-afd0-d7e8ad0762e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:18:08.754764 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:08.754650 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-kube-api-access-s6gjt" pod "f456c7bf-4939-4c59-afd0-d7e8ad0762e8" (UID: "f456c7bf-4939-4c59-afd0-d7e8ad0762e8"). InnerVolumeSpecName "kube-api-access-s6gjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:18:08.853892 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:08.853820 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6gjt\" (UniqueName: \"kubernetes.io/projected/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-kube-api-access-s6gjt\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:08.853892 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:08.853845 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:08.853892 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:08.853855 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f456c7bf-4939-4c59-afd0-d7e8ad0762e8-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:09.485355 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.485329 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 03 00:18:09.534015 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.533932 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" Nov 03 00:18:09.534015 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.533955 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd" event={"ID":"f456c7bf-4939-4c59-afd0-d7e8ad0762e8","Type":"ContainerDied","Data":"f6734a5d7738c9727fdbd694ad04b32ec7b269ff8572affb3a19c523c7607f50"} Nov 03 00:18:09.534015 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.534001 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6734a5d7738c9727fdbd694ad04b32ec7b269ff8572affb3a19c523c7607f50" Nov 03 00:18:09.535709 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.535686 5672 generic.go:356] "Generic (PLEG): container finished" podID="33e415f6-4842-4991-bd47-03d50d0354bf" containerID="92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009" exitCode=0 Nov 03 00:18:09.535805 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.535737 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" event={"ID":"33e415f6-4842-4991-bd47-03d50d0354bf","Type":"ContainerDied","Data":"92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009"} Nov 03 00:18:09.535805 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.535769 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" event={"ID":"33e415f6-4842-4991-bd47-03d50d0354bf","Type":"ContainerDied","Data":"213b3c3e4eda179e0c2cd3f723dd493dc6711a073e08de0cd4140b867735b0dd"} Nov 03 00:18:09.535805 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.535792 5672 scope.go:122] "RemoveContainer" containerID="92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009" Nov 03 00:18:09.535936 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.535737 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg" Nov 03 00:18:09.545610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.545592 5672 scope.go:122] "RemoveContainer" containerID="aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90" Nov 03 00:18:09.554389 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.554369 5672 scope.go:122] "RemoveContainer" containerID="92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009" Nov 03 00:18:09.554656 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:18:09.554639 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009\": container with ID starting with 92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009 not found: ID does not exist" containerID="92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009" Nov 03 00:18:09.554699 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.554668 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009"} err="failed to get container status \"92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009\": rpc error: code = NotFound desc = could not find container \"92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009\": container with ID starting with 92ee62300e32616fec74b6c70df2cefbd64cb6d408a5c5f6d5d9085dfa8b7009 not found: ID does not exist" Nov 03 00:18:09.554699 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.554685 5672 scope.go:122] "RemoveContainer" containerID="aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90" Nov 03 00:18:09.554952 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:18:09.554933 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90\": container with ID starting with aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90 not found: ID does not exist" containerID="aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90" Nov 03 00:18:09.554993 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.554956 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90"} err="failed to get container status \"aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90\": rpc error: code = NotFound desc = could not find container \"aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90\": container with ID starting with aa74618475cbc7fffad547f87f93eca423c78a7cd8ac3bb3691c1fba17e6fd90 not found: ID does not exist" Nov 03 00:18:09.560344 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560322 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-socket\") pod \"33e415f6-4842-4991-bd47-03d50d0354bf\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " Nov 03 00:18:09.560427 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560385 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-credential-socket\") pod \"33e415f6-4842-4991-bd47-03d50d0354bf\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " Nov 03 00:18:09.560479 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560444 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-certs\") pod \"33e415f6-4842-4991-bd47-03d50d0354bf\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " Nov 03 00:18:09.560523 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560478 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-data\") pod \"33e415f6-4842-4991-bd47-03d50d0354bf\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " Nov 03 00:18:09.560523 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560516 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33e415f6-4842-4991-bd47-03d50d0354bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e415f6-4842-4991-bd47-03d50d0354bf-istio-podinfo\") pod \"33e415f6-4842-4991-bd47-03d50d0354bf\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " Nov 03 00:18:09.560599 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560575 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-istio-token\" (UniqueName: \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-istio-token\") pod \"33e415f6-4842-4991-bd47-03d50d0354bf\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " Nov 03 00:18:09.560599 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560574 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-socket" pod "33e415f6-4842-4991-bd47-03d50d0354bf" (UID: "33e415f6-4842-4991-bd47-03d50d0354bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:09.560679 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560605 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e415f6-4842-4991-bd47-03d50d0354bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e415f6-4842-4991-bd47-03d50d0354bf-istiod-ca-cert\") pod \"33e415f6-4842-4991-bd47-03d50d0354bf\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " Nov 03 00:18:09.560679 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560641 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-envoy\") pod \"33e415f6-4842-4991-bd47-03d50d0354bf\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " Nov 03 00:18:09.560777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560689 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-kube-api-access-m446m\" (UniqueName: \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-kube-api-access-m446m\") pod \"33e415f6-4842-4991-bd47-03d50d0354bf\" (UID: \"33e415f6-4842-4991-bd47-03d50d0354bf\") " Nov 03 00:18:09.560777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560702 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-credential-socket" pod "33e415f6-4842-4991-bd47-03d50d0354bf" (UID: "33e415f6-4842-4991-bd47-03d50d0354bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:09.560777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560742 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-certs" pod "33e415f6-4842-4991-bd47-03d50d0354bf" (UID: "33e415f6-4842-4991-bd47-03d50d0354bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:09.560970 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560954 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:09.561047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560974 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:09.561047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560995 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:09.561047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.560969 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-data" pod "33e415f6-4842-4991-bd47-03d50d0354bf" (UID: "33e415f6-4842-4991-bd47-03d50d0354bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:09.561228 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.561113 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e415f6-4842-4991-bd47-03d50d0354bf-istiod-ca-cert" pod "33e415f6-4842-4991-bd47-03d50d0354bf" (UID: "33e415f6-4842-4991-bd47-03d50d0354bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:18:09.562728 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.562708 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33e415f6-4842-4991-bd47-03d50d0354bf-istio-podinfo" pod "33e415f6-4842-4991-bd47-03d50d0354bf" (UID: "33e415f6-4842-4991-bd47-03d50d0354bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:18:09.562870 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.562850 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-istio-token" pod "33e415f6-4842-4991-bd47-03d50d0354bf" (UID: "33e415f6-4842-4991-bd47-03d50d0354bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:18:09.563130 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.563111 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-envoy" pod "33e415f6-4842-4991-bd47-03d50d0354bf" (UID: "33e415f6-4842-4991-bd47-03d50d0354bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:09.563210 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.563159 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-kube-api-access-m446m" pod "33e415f6-4842-4991-bd47-03d50d0354bf" (UID: "33e415f6-4842-4991-bd47-03d50d0354bf"). InnerVolumeSpecName "kube-api-access-m446m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:18:09.662181 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.662129 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:09.662348 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.662185 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e415f6-4842-4991-bd47-03d50d0354bf-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:09.662348 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.662202 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:09.662348 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.662216 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m446m\" (UniqueName: \"kubernetes.io/projected/33e415f6-4842-4991-bd47-03d50d0354bf-kube-api-access-m446m\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:09.662348 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.662230 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e415f6-4842-4991-bd47-03d50d0354bf-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:09.662348 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.662243 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e415f6-4842-4991-bd47-03d50d0354bf-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:09.704224 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.704193 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm"] Nov 03 00:18:09.717680 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:09.717652 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4"] Nov 03 00:18:10.086948 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:10.086903 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg"] Nov 03 00:18:10.090464 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:10.090434 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ca60f-openshift-default-6ffff56c89-dctdg"] Nov 03 00:18:10.712039 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:10.712006 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" path="/var/lib/kubelet/pods/33e415f6-4842-4991-bd47-03d50d0354bf/volumes" Nov 03 00:18:14.056942 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:14.056903 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n"] Nov 03 00:18:14.103144 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:14.103109 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ea45ea-mqb7n"] Nov 03 00:18:14.711955 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:14.711920 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c518153a-74d5-495f-a977-65e65885c42c" path="/var/lib/kubelet/pods/c518153a-74d5-495f-a977-65e65885c42c/volumes" Nov 03 00:18:17.750333 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:17.750286 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns"] Nov 03 00:18:17.750927 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:17.750513 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" containerID="cri-o://dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c" gracePeriod=30 Nov 03 00:18:18.501613 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:18.501574 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-57kns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:18:18.501796 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:18.501636 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:18:21.931519 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:21.931482 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j"] Nov 03 00:18:21.931935 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:21.931878 5672 cpu_manager.go:414] "containerMap: removing container" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" Nov 03 00:18:21.931935 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:21.931890 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" Nov 03 00:18:21.931935 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:21.931921 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f456c7bf-4939-4c59-afd0-d7e8ad0762e8" containerName="revoke-keys" Nov 03 00:18:21.931935 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:21.931926 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f456c7bf-4939-4c59-afd0-d7e8ad0762e8" containerName="revoke-keys" Nov 03 00:18:21.932080 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:21.931984 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" Nov 03 00:18:21.932080 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:21.931993 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" Nov 03 00:18:21.932080 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:21.932005 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f456c7bf-4939-4c59-afd0-d7e8ad0762e8" containerName="revoke-keys" Nov 03 00:18:21.936945 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:21.936918 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:21.988764 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:21.988724 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:18:21.991943 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:21.991913 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j"] Nov 03 00:18:22.035063 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.035029 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:18:22.035389 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.035360 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:18:22.035389 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.035372 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:18:22.086121 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.086069 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef4c8094-7dr8j\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:22.086121 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.086121 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef4c8094-7dr8j\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:22.086387 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.086250 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbc8h\" (UniqueName: \"kubernetes.io/projected/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-kube-api-access-bbc8h\") pod \"maas-api-revoke-sub-ef4c8094-7dr8j\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:22.187727 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.187623 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbc8h\" (UniqueName: \"kubernetes.io/projected/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-kube-api-access-bbc8h\") pod \"maas-api-revoke-sub-ef4c8094-7dr8j\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:22.187928 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.187760 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef4c8094-7dr8j\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:22.187928 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.187795 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef4c8094-7dr8j\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:22.188635 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.188611 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef4c8094-7dr8j\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:22.190543 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.190520 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef4c8094-7dr8j\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:22.242957 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.242921 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbc8h\" (UniqueName: \"kubernetes.io/projected/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-kube-api-access-bbc8h\") pod \"maas-api-revoke-sub-ef4c8094-7dr8j\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:22.249732 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.249702 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:22.625050 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.625021 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j"] Nov 03 00:18:22.627125 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:18:22.627092 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40d50c3_bdc7_443d_9c6b_0b438a20a6b7.slice/crio-6f75ce13e46b0d96cf8ccd6a4c12129a07f2a6f4090256713d5c19aba83a1c6b WatchSource:0}: Error finding container 6f75ce13e46b0d96cf8ccd6a4c12129a07f2a6f4090256713d5c19aba83a1c6b: Status 404 returned error can't find the container with id 6f75ce13e46b0d96cf8ccd6a4c12129a07f2a6f4090256713d5c19aba83a1c6b Nov 03 00:18:22.988414 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:22.988385 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 03 00:18:23.095949 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.095910 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-podinfo\") pod \"84720bad-6f3c-48da-9fec-8dc56a7de315\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " Nov 03 00:18:23.096191 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.095970 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-token\" (UniqueName: \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-token\") pod \"84720bad-6f3c-48da-9fec-8dc56a7de315\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " Nov 03 00:18:23.096191 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096001 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-data\") pod \"84720bad-6f3c-48da-9fec-8dc56a7de315\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " Nov 03 00:18:23.096191 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096028 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-envoy\") pod \"84720bad-6f3c-48da-9fec-8dc56a7de315\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " Nov 03 00:18:23.096191 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096045 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-kube-api-access-fk4hd\" (UniqueName: \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-kube-api-access-fk4hd\") pod \"84720bad-6f3c-48da-9fec-8dc56a7de315\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " Nov 03 00:18:23.096191 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096084 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-credential-socket\") pod \"84720bad-6f3c-48da-9fec-8dc56a7de315\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " Nov 03 00:18:23.096561 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096224 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-certs\") pod \"84720bad-6f3c-48da-9fec-8dc56a7de315\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " Nov 03 00:18:23.096561 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096284 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-socket\") pod \"84720bad-6f3c-48da-9fec-8dc56a7de315\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " Nov 03 00:18:23.096561 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096311 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84720bad-6f3c-48da-9fec-8dc56a7de315-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84720bad-6f3c-48da-9fec-8dc56a7de315-istiod-ca-cert\") pod \"84720bad-6f3c-48da-9fec-8dc56a7de315\" (UID: \"84720bad-6f3c-48da-9fec-8dc56a7de315\") " Nov 03 00:18:23.096561 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096460 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-credential-socket" pod "84720bad-6f3c-48da-9fec-8dc56a7de315" (UID: "84720bad-6f3c-48da-9fec-8dc56a7de315"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:23.096561 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096486 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-data" pod "84720bad-6f3c-48da-9fec-8dc56a7de315" (UID: "84720bad-6f3c-48da-9fec-8dc56a7de315"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:23.096796 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096583 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-certs" pod "84720bad-6f3c-48da-9fec-8dc56a7de315" (UID: "84720bad-6f3c-48da-9fec-8dc56a7de315"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:23.096796 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096773 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84720bad-6f3c-48da-9fec-8dc56a7de315-istiod-ca-cert" pod "84720bad-6f3c-48da-9fec-8dc56a7de315" (UID: "84720bad-6f3c-48da-9fec-8dc56a7de315"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:18:23.096918 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096759 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:23.096918 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096879 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:23.096918 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096889 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-socket" pod "84720bad-6f3c-48da-9fec-8dc56a7de315" (UID: "84720bad-6f3c-48da-9fec-8dc56a7de315"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:23.096918 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.096897 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:23.098771 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.098742 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-envoy" pod "84720bad-6f3c-48da-9fec-8dc56a7de315" (UID: "84720bad-6f3c-48da-9fec-8dc56a7de315"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:23.098866 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.098787 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-kube-api-access-fk4hd" pod "84720bad-6f3c-48da-9fec-8dc56a7de315" (UID: "84720bad-6f3c-48da-9fec-8dc56a7de315"). InnerVolumeSpecName "kube-api-access-fk4hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:18:23.099002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.098980 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-token" pod "84720bad-6f3c-48da-9fec-8dc56a7de315" (UID: "84720bad-6f3c-48da-9fec-8dc56a7de315"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:18:23.099060 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.098996 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-podinfo" pod "84720bad-6f3c-48da-9fec-8dc56a7de315" (UID: "84720bad-6f3c-48da-9fec-8dc56a7de315"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:18:23.198309 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.198102 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:23.198309 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.198141 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84720bad-6f3c-48da-9fec-8dc56a7de315-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:23.198309 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.198178 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:23.198309 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.198195 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:23.198309 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.198209 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84720bad-6f3c-48da-9fec-8dc56a7de315-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:23.198309 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.198228 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk4hd\" (UniqueName: \"kubernetes.io/projected/84720bad-6f3c-48da-9fec-8dc56a7de315-kube-api-access-fk4hd\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:23.625272 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.625231 5672 generic.go:356] "Generic (PLEG): container finished" podID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerID="dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c" exitCode=0 Nov 03 00:18:23.625472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.625280 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" Nov 03 00:18:23.625472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.625316 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" event={"ID":"84720bad-6f3c-48da-9fec-8dc56a7de315","Type":"ContainerDied","Data":"dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c"} Nov 03 00:18:23.625472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.625372 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns" event={"ID":"84720bad-6f3c-48da-9fec-8dc56a7de315","Type":"ContainerDied","Data":"6a3076881ee9c6879e54413bf9ea21f919f207e8ba0e84c5907f2708bedf30cc"} Nov 03 00:18:23.625472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.625403 5672 scope.go:122] "RemoveContainer" containerID="dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c" Nov 03 00:18:23.627337 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.627304 5672 generic.go:356] "Generic (PLEG): container finished" podID="e40d50c3-bdc7-443d-9c6b-0b438a20a6b7" containerID="9c8dcb7ee4d27ecb8d5765bdd79f021266b93ab1255e14efc3c37604288a4fa5" exitCode=0 Nov 03 00:18:23.627466 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.627403 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" event={"ID":"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7","Type":"ContainerDied","Data":"9c8dcb7ee4d27ecb8d5765bdd79f021266b93ab1255e14efc3c37604288a4fa5"} Nov 03 00:18:23.627534 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.627479 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" event={"ID":"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7","Type":"ContainerStarted","Data":"6f75ce13e46b0d96cf8ccd6a4c12129a07f2a6f4090256713d5c19aba83a1c6b"} Nov 03 00:18:23.637292 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.637266 5672 scope.go:122] "RemoveContainer" containerID="dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c" Nov 03 00:18:23.637649 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:18:23.637633 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c\": container with ID starting with dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c not found: ID does not exist" containerID="dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c" Nov 03 00:18:23.637701 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.637658 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c"} err="failed to get container status \"dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c\": rpc error: code = NotFound desc = could not find container \"dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c\": container with ID starting with dece27cf3d36f1e8bec8097e2c44211acc75cc26d13e705de53740205b456a9c not found: ID does not exist" Nov 03 00:18:23.797269 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.797127 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns"] Nov 03 00:18:23.806312 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:23.806271 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-57kns"] Nov 03 00:18:24.713628 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:24.713587 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" path="/var/lib/kubelet/pods/84720bad-6f3c-48da-9fec-8dc56a7de315/volumes" Nov 03 00:18:24.753081 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:24.752890 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" containerID="cri-o://b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a" gracePeriod=45 Nov 03 00:18:24.753081 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:24.753055 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" podUID="a4fd478e-361d-479a-9703-bb9a72ef8c17" containerName="main" containerID="cri-o://87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a" gracePeriod=45 Nov 03 00:18:24.903247 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:24.901377 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:24.934851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:24.934813 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": dial tcp 10.134.0.45:8000: connect: connection refused" Nov 03 00:18:25.016337 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.016304 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 03 00:18:25.016788 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.016761 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-kube-api-access-bbc8h\" (UniqueName: \"kubernetes.io/projected/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-kube-api-access-bbc8h\") pod \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " Nov 03 00:18:25.016950 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.016842 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-xks-ca\") pod \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " Nov 03 00:18:25.017014 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.016977 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-openshift-ca\") pod \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\" (UID: \"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7\") " Nov 03 00:18:25.017445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.017421 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-openshift-ca" pod "e40d50c3-bdc7-443d-9c6b-0b438a20a6b7" (UID: "e40d50c3-bdc7-443d-9c6b-0b438a20a6b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:18:25.021925 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.020166 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-xks-ca" pod "e40d50c3-bdc7-443d-9c6b-0b438a20a6b7" (UID: "e40d50c3-bdc7-443d-9c6b-0b438a20a6b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:18:25.021925 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.020361 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-kube-api-access-bbc8h" pod "e40d50c3-bdc7-443d-9c6b-0b438a20a6b7" (UID: "e40d50c3-bdc7-443d-9c6b-0b438a20a6b7"). InnerVolumeSpecName "kube-api-access-bbc8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:18:25.039785 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.039750 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 03 00:18:25.118363 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.118323 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4fd478e-361d-479a-9703-bb9a72ef8c17-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4fd478e-361d-479a-9703-bb9a72ef8c17-tls-certs\") pod \"a4fd478e-361d-479a-9703-bb9a72ef8c17\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " Nov 03 00:18:25.118565 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.118388 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-model-cache\") pod \"a4fd478e-361d-479a-9703-bb9a72ef8c17\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " Nov 03 00:18:25.118565 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.118444 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-tmp-dir\") pod \"a4fd478e-361d-479a-9703-bb9a72ef8c17\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " Nov 03 00:18:25.118565 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.118486 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-home\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-home\") pod \"a4fd478e-361d-479a-9703-bb9a72ef8c17\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " Nov 03 00:18:25.118751 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.118605 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-dshm\") pod \"a4fd478e-361d-479a-9703-bb9a72ef8c17\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " Nov 03 00:18:25.118751 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.118647 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-model-cache" pod "a4fd478e-361d-479a-9703-bb9a72ef8c17" (UID: "a4fd478e-361d-479a-9703-bb9a72ef8c17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.118751 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.118667 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4fd478e-361d-479a-9703-bb9a72ef8c17-kube-api-access-gk5hv\" (UniqueName: \"kubernetes.io/projected/a4fd478e-361d-479a-9703-bb9a72ef8c17-kube-api-access-gk5hv\") pod \"a4fd478e-361d-479a-9703-bb9a72ef8c17\" (UID: \"a4fd478e-361d-479a-9703-bb9a72ef8c17\") " Nov 03 00:18:25.118916 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.118769 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-tmp-dir" pod "a4fd478e-361d-479a-9703-bb9a72ef8c17" (UID: "a4fd478e-361d-479a-9703-bb9a72ef8c17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.118916 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.118863 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-home" pod "a4fd478e-361d-479a-9703-bb9a72ef8c17" (UID: "a4fd478e-361d-479a-9703-bb9a72ef8c17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.119074 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.119052 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbc8h\" (UniqueName: \"kubernetes.io/projected/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-kube-api-access-bbc8h\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.119137 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.119081 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.119137 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.119098 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.119137 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.119113 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.119137 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.119127 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.119302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.119143 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.121007 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.120975 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-dshm" pod "a4fd478e-361d-479a-9703-bb9a72ef8c17" (UID: "a4fd478e-361d-479a-9703-bb9a72ef8c17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.121269 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.121100 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fd478e-361d-479a-9703-bb9a72ef8c17-kube-api-access-gk5hv" pod "a4fd478e-361d-479a-9703-bb9a72ef8c17" (UID: "a4fd478e-361d-479a-9703-bb9a72ef8c17"). InnerVolumeSpecName "kube-api-access-gk5hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:18:25.121369 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.121293 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4fd478e-361d-479a-9703-bb9a72ef8c17-tls-certs" pod "a4fd478e-361d-479a-9703-bb9a72ef8c17" (UID: "a4fd478e-361d-479a-9703-bb9a72ef8c17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:18:25.219891 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.219857 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-model-cache\") pod \"04740004-3f15-406b-a220-8926e141df5e\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " Nov 03 00:18:25.220093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.219910 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-dshm\") pod \"04740004-3f15-406b-a220-8926e141df5e\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " Nov 03 00:18:25.220093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.219944 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-home\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-home\") pod \"04740004-3f15-406b-a220-8926e141df5e\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " Nov 03 00:18:25.220093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.219970 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-tmp-dir\") pod \"04740004-3f15-406b-a220-8926e141df5e\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " Nov 03 00:18:25.220093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.220018 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04740004-3f15-406b-a220-8926e141df5e-kube-api-access-6kjnh\" (UniqueName: \"kubernetes.io/projected/04740004-3f15-406b-a220-8926e141df5e-kube-api-access-6kjnh\") pod \"04740004-3f15-406b-a220-8926e141df5e\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " Nov 03 00:18:25.220333 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.220135 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04740004-3f15-406b-a220-8926e141df5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/04740004-3f15-406b-a220-8926e141df5e-tls-certs\") pod \"04740004-3f15-406b-a220-8926e141df5e\" (UID: \"04740004-3f15-406b-a220-8926e141df5e\") " Nov 03 00:18:25.220333 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.220240 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-model-cache" pod "04740004-3f15-406b-a220-8926e141df5e" (UID: "04740004-3f15-406b-a220-8926e141df5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.220333 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.220257 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-home" pod "04740004-3f15-406b-a220-8926e141df5e" (UID: "04740004-3f15-406b-a220-8926e141df5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.220333 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.220280 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-tmp-dir" pod "04740004-3f15-406b-a220-8926e141df5e" (UID: "04740004-3f15-406b-a220-8926e141df5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.220506 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.220487 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4fd478e-361d-479a-9703-bb9a72ef8c17-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.220568 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.220509 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk5hv\" (UniqueName: \"kubernetes.io/projected/a4fd478e-361d-479a-9703-bb9a72ef8c17-kube-api-access-gk5hv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.220568 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.220518 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4fd478e-361d-479a-9703-bb9a72ef8c17-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.220568 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.220528 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.220568 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.220538 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.220568 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.220546 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.222111 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.222084 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-dshm" pod "04740004-3f15-406b-a220-8926e141df5e" (UID: "04740004-3f15-406b-a220-8926e141df5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:18:25.222267 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.222137 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04740004-3f15-406b-a220-8926e141df5e-tls-certs" pod "04740004-3f15-406b-a220-8926e141df5e" (UID: "04740004-3f15-406b-a220-8926e141df5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:18:25.222267 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.222185 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04740004-3f15-406b-a220-8926e141df5e-kube-api-access-6kjnh" pod "04740004-3f15-406b-a220-8926e141df5e" (UID: "04740004-3f15-406b-a220-8926e141df5e"). InnerVolumeSpecName "kube-api-access-6kjnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:18:25.321884 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.321836 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04740004-3f15-406b-a220-8926e141df5e-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.321884 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.321871 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kjnh\" (UniqueName: \"kubernetes.io/projected/04740004-3f15-406b-a220-8926e141df5e-kube-api-access-6kjnh\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.321884 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.321881 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04740004-3f15-406b-a220-8926e141df5e-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:18:25.643170 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.643053 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" Nov 03 00:18:25.643170 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.643062 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j" event={"ID":"e40d50c3-bdc7-443d-9c6b-0b438a20a6b7","Type":"ContainerDied","Data":"6f75ce13e46b0d96cf8ccd6a4c12129a07f2a6f4090256713d5c19aba83a1c6b"} Nov 03 00:18:25.643170 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.643113 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f75ce13e46b0d96cf8ccd6a4c12129a07f2a6f4090256713d5c19aba83a1c6b" Nov 03 00:18:25.645186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.644552 5672 generic.go:356] "Generic (PLEG): container finished" podID="a4fd478e-361d-479a-9703-bb9a72ef8c17" containerID="87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a" exitCode=0 Nov 03 00:18:25.645186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.644630 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" Nov 03 00:18:25.645186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.644719 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" event={"ID":"a4fd478e-361d-479a-9703-bb9a72ef8c17","Type":"ContainerDied","Data":"87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a"} Nov 03 00:18:25.645186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.644751 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm" event={"ID":"a4fd478e-361d-479a-9703-bb9a72ef8c17","Type":"ContainerDied","Data":"9e7bf34f344f0b5c641e99b87cd70543f102822f8e2ca07c79871052bab838ec"} Nov 03 00:18:25.645186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.644773 5672 scope.go:122] "RemoveContainer" containerID="87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a" Nov 03 00:18:25.664878 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.661209 5672 generic.go:356] "Generic (PLEG): container finished" podID="04740004-3f15-406b-a220-8926e141df5e" containerID="b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a" exitCode=0 Nov 03 00:18:25.664878 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.661273 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" event={"ID":"04740004-3f15-406b-a220-8926e141df5e","Type":"ContainerDied","Data":"b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a"} Nov 03 00:18:25.664878 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.661310 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" event={"ID":"04740004-3f15-406b-a220-8926e141df5e","Type":"ContainerDied","Data":"b775b51d048c60eda954f59eae49962cddefb765d87ce3cdd405f9008eb2df1e"} Nov 03 00:18:25.664878 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.661382 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4" Nov 03 00:18:25.668397 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.668371 5672 scope.go:122] "RemoveContainer" containerID="87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a" Nov 03 00:18:25.668922 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:18:25.668898 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a\": container with ID starting with 87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a not found: ID does not exist" containerID="87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a" Nov 03 00:18:25.669020 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.668934 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a"} err="failed to get container status \"87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a\": rpc error: code = NotFound desc = could not find container \"87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a\": container with ID starting with 87996a91c79a76fde7bfd8914545f868873b7d4d1790c38ffc594f7ceb235e3a not found: ID does not exist" Nov 03 00:18:25.669020 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.668961 5672 scope.go:122] "RemoveContainer" containerID="b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a" Nov 03 00:18:25.680196 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.680143 5672 scope.go:122] "RemoveContainer" containerID="b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a" Nov 03 00:18:25.680500 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:18:25.680479 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a\": container with ID starting with b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a not found: ID does not exist" containerID="b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a" Nov 03 00:18:25.680568 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.680512 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a"} err="failed to get container status \"b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a\": rpc error: code = NotFound desc = could not find container \"b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a\": container with ID starting with b200b6b34569e8f16d928820c326c71cddd3e51f396a5e4d0a921bf0b97fef1a not found: ID does not exist" Nov 03 00:18:25.801361 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.801297 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4"] Nov 03 00:18:25.847339 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.847305 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/premium-w3-14b321-kserve-bbccc9f74-z7bh4"] Nov 03 00:18:25.957538 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.957453 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm"] Nov 03 00:18:25.961700 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:25.961670 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/distinct-w3-14b321-kserve-9b86f6dfc-2ltgm"] Nov 03 00:18:26.021554 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:18:26.021517 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:18:26.713821 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:26.713785 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04740004-3f15-406b-a220-8926e141df5e" path="/var/lib/kubelet/pods/04740004-3f15-406b-a220-8926e141df5e/volumes" Nov 03 00:18:26.714446 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:26.714423 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4fd478e-361d-479a-9703-bb9a72ef8c17" path="/var/lib/kubelet/pods/a4fd478e-361d-479a-9703-bb9a72ef8c17/volumes" Nov 03 00:18:29.722875 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:29.722837 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:18:30.025791 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:30.025708 5672 scope.go:122] "RemoveContainer" containerID="b9e607988bdfbfcc8dafc6defdd873ee8f1cfb9d6f4d55f95d367a8719c700e0" Nov 03 00:18:30.035733 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:18:30.035714 5672 scope.go:122] "RemoveContainer" containerID="cba68e4c4bdda5b3690349169705c58e15b57416e2f58387ddf564eeed3cb55d" Nov 03 00:18:56.021118 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:18:56.021077 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:19:08.874016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:08.873932 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:19:25.975246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:25.975208 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:19:26.020638 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:19:26.020599 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:19:27.740299 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:27.740260 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh"] Nov 03 00:19:30.097190 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:30.097134 5672 scope.go:122] "RemoveContainer" containerID="8fce1acef2a8a5b0be56c2c881d9cdc2991016037c9cceefc5646030c6426b9f" Nov 03 00:19:30.107395 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:30.107371 5672 scope.go:122] "RemoveContainer" containerID="7f6331bfaf730ed215a3cd77fc461add590b29d18030e954950060193d6c8a8d" Nov 03 00:19:32.741206 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:32.741163 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" podUID="612b0292-9195-4729-bcfb-44ea8d80a9c0" containerName="payload-processing" containerID="cri-o://82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782" gracePeriod=25 Nov 03 00:19:35.021027 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:35.020989 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" podUID="612b0292-9195-4729-bcfb-44ea8d80a9c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:19:41.976049 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:41.976009 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:19:45.021122 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:45.021082 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" podUID="612b0292-9195-4729-bcfb-44ea8d80a9c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:19:51.223223 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223186 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg"] Nov 03 00:19:51.223632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223570 5672 cpu_manager.go:414] "containerMap: removing container" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" Nov 03 00:19:51.223632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223582 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e415f6-4842-4991-bd47-03d50d0354bf" containerName="istio-proxy" Nov 03 00:19:51.223632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223596 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a4fd478e-361d-479a-9703-bb9a72ef8c17" containerName="main" Nov 03 00:19:51.223632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223602 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4fd478e-361d-479a-9703-bb9a72ef8c17" containerName="main" Nov 03 00:19:51.223632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223613 5672 cpu_manager.go:414] "containerMap: removing container" podUID="e40d50c3-bdc7-443d-9c6b-0b438a20a6b7" containerName="revoke-keys" Nov 03 00:19:51.223632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223618 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40d50c3-bdc7-443d-9c6b-0b438a20a6b7" containerName="revoke-keys" Nov 03 00:19:51.223632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223631 5672 cpu_manager.go:414] "containerMap: removing container" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" Nov 03 00:19:51.223632 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223635 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" Nov 03 00:19:51.223906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223643 5672 cpu_manager.go:414] "containerMap: removing container" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" Nov 03 00:19:51.223906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223648 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" Nov 03 00:19:51.223906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223707 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4fd478e-361d-479a-9703-bb9a72ef8c17" containerName="main" Nov 03 00:19:51.223906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223717 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40d50c3-bdc7-443d-9c6b-0b438a20a6b7" containerName="revoke-keys" Nov 03 00:19:51.223906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223724 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="04740004-3f15-406b-a220-8926e141df5e" containerName="main" Nov 03 00:19:51.223906 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.223729 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="84720bad-6f3c-48da-9fec-8dc56a7de315" containerName="istio-proxy" Nov 03 00:19:51.226972 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.226952 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:51.322826 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.322753 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:19:51.322826 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.322756 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:19:51.322826 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.322815 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:19:51.323106 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.322842 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:19:51.323106 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.323035 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sc5c\" (UniqueName: \"kubernetes.io/projected/d8807f9e-1791-40cb-887a-27c3f6c529a7-kube-api-access-6sc5c\") pod \"maas-api-revoke-sub-b6cfdc2c-m9kxg\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:51.323213 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.323183 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6cfdc2c-m9kxg\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:51.323251 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.323235 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6cfdc2c-m9kxg\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:51.325215 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.325195 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg"] Nov 03 00:19:51.424336 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.424301 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6cfdc2c-m9kxg\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:51.424336 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.424336 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6cfdc2c-m9kxg\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:51.424615 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.424384 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sc5c\" (UniqueName: \"kubernetes.io/projected/d8807f9e-1791-40cb-887a-27c3f6c529a7-kube-api-access-6sc5c\") pod \"maas-api-revoke-sub-b6cfdc2c-m9kxg\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:51.424978 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.424954 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6cfdc2c-m9kxg\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:51.426822 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.426790 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6cfdc2c-m9kxg\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:51.479454 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.479375 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sc5c\" (UniqueName: \"kubernetes.io/projected/d8807f9e-1791-40cb-887a-27c3f6c529a7-kube-api-access-6sc5c\") pod \"maas-api-revoke-sub-b6cfdc2c-m9kxg\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:51.539542 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.539511 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:51.930415 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:51.930392 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg"] Nov 03 00:19:51.932018 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:19:51.931987 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8807f9e_1791_40cb_887a_27c3f6c529a7.slice/crio-e6f25532b716115b5da50374cd2cc5182ea6b2b7f913a86d5216d7cd79b59ae1 WatchSource:0}: Error finding container e6f25532b716115b5da50374cd2cc5182ea6b2b7f913a86d5216d7cd79b59ae1: Status 404 returned error can't find the container with id e6f25532b716115b5da50374cd2cc5182ea6b2b7f913a86d5216d7cd79b59ae1 Nov 03 00:19:52.156406 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:52.156368 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" event={"ID":"d8807f9e-1791-40cb-887a-27c3f6c529a7","Type":"ContainerStarted","Data":"e6f25532b716115b5da50374cd2cc5182ea6b2b7f913a86d5216d7cd79b59ae1"} Nov 03 00:19:53.012626 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.012547 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 03 00:19:53.044950 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.044910 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c"] Nov 03 00:19:53.094739 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.094683 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422ce4fb-m6t4c"] Nov 03 00:19:53.140224 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.140132 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612b0292-9195-4729-bcfb-44ea8d80a9c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612b0292-9195-4729-bcfb-44ea8d80a9c0-plugins-config-volume\") pod \"612b0292-9195-4729-bcfb-44ea8d80a9c0\" (UID: \"612b0292-9195-4729-bcfb-44ea8d80a9c0\") " Nov 03 00:19:53.140224 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.140202 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612b0292-9195-4729-bcfb-44ea8d80a9c0-kube-api-access-b9wgv\" (UniqueName: \"kubernetes.io/projected/612b0292-9195-4729-bcfb-44ea8d80a9c0-kube-api-access-b9wgv\") pod \"612b0292-9195-4729-bcfb-44ea8d80a9c0\" (UID: \"612b0292-9195-4729-bcfb-44ea8d80a9c0\") " Nov 03 00:19:53.140790 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.140758 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612b0292-9195-4729-bcfb-44ea8d80a9c0-plugins-config-volume" pod "612b0292-9195-4729-bcfb-44ea8d80a9c0" (UID: "612b0292-9195-4729-bcfb-44ea8d80a9c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:53.142544 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.142514 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612b0292-9195-4729-bcfb-44ea8d80a9c0-kube-api-access-b9wgv" pod "612b0292-9195-4729-bcfb-44ea8d80a9c0" (UID: "612b0292-9195-4729-bcfb-44ea8d80a9c0"). InnerVolumeSpecName "kube-api-access-b9wgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:53.163559 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.163523 5672 generic.go:356] "Generic (PLEG): container finished" podID="612b0292-9195-4729-bcfb-44ea8d80a9c0" containerID="82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782" exitCode=0 Nov 03 00:19:53.163742 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.163588 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" Nov 03 00:19:53.163742 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.163598 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" event={"ID":"612b0292-9195-4729-bcfb-44ea8d80a9c0","Type":"ContainerDied","Data":"82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782"} Nov 03 00:19:53.163742 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.163704 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh" event={"ID":"612b0292-9195-4729-bcfb-44ea8d80a9c0","Type":"ContainerDied","Data":"e1d9641bb7031cffe2296c2e23fef9867eb566951f9f203b4acafd571080d69e"} Nov 03 00:19:53.163921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.163751 5672 scope.go:122] "RemoveContainer" containerID="82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782" Nov 03 00:19:53.165469 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.165439 5672 generic.go:356] "Generic (PLEG): container finished" podID="d8807f9e-1791-40cb-887a-27c3f6c529a7" containerID="d7f864e793fb9d26030fba5b86cc1e3f8e25a8b9ad4a8415f52057aeef7de740" exitCode=0 Nov 03 00:19:53.165469 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.165477 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" event={"ID":"d8807f9e-1791-40cb-887a-27c3f6c529a7","Type":"ContainerDied","Data":"d7f864e793fb9d26030fba5b86cc1e3f8e25a8b9ad4a8415f52057aeef7de740"} Nov 03 00:19:53.173946 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.173924 5672 scope.go:122] "RemoveContainer" containerID="82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782" Nov 03 00:19:53.174284 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:19:53.174265 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782\": container with ID starting with 82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782 not found: ID does not exist" containerID="82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782" Nov 03 00:19:53.174366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.174295 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782"} err="failed to get container status \"82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782\": rpc error: code = NotFound desc = could not find container \"82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782\": container with ID starting with 82a429db877486283caf9774f59aa68a7fdcbc4fb3106deb4cf69282c4d92782 not found: ID does not exist" Nov 03 00:19:53.241619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.241581 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612b0292-9195-4729-bcfb-44ea8d80a9c0-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:53.241619 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.241610 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9wgv\" (UniqueName: \"kubernetes.io/projected/612b0292-9195-4729-bcfb-44ea8d80a9c0-kube-api-access-b9wgv\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:53.429612 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.429533 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh"] Nov 03 00:19:53.435351 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:53.435306 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a51378-745d84f5d9-nt5bh"] Nov 03 00:19:54.315878 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.315850 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:54.453631 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.453532 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-xks-ca\") pod \"d8807f9e-1791-40cb-887a-27c3f6c529a7\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " Nov 03 00:19:54.453631 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.453588 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8807f9e-1791-40cb-887a-27c3f6c529a7-kube-api-access-6sc5c\" (UniqueName: \"kubernetes.io/projected/d8807f9e-1791-40cb-887a-27c3f6c529a7-kube-api-access-6sc5c\") pod \"d8807f9e-1791-40cb-887a-27c3f6c529a7\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " Nov 03 00:19:54.453631 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.453627 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-openshift-ca\") pod \"d8807f9e-1791-40cb-887a-27c3f6c529a7\" (UID: \"d8807f9e-1791-40cb-887a-27c3f6c529a7\") " Nov 03 00:19:54.454161 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.454112 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-openshift-ca" pod "d8807f9e-1791-40cb-887a-27c3f6c529a7" (UID: "d8807f9e-1791-40cb-887a-27c3f6c529a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:54.455959 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.455926 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-xks-ca" pod "d8807f9e-1791-40cb-887a-27c3f6c529a7" (UID: "d8807f9e-1791-40cb-887a-27c3f6c529a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:54.456348 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.456318 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8807f9e-1791-40cb-887a-27c3f6c529a7-kube-api-access-6sc5c" pod "d8807f9e-1791-40cb-887a-27c3f6c529a7" (UID: "d8807f9e-1791-40cb-887a-27c3f6c529a7"). InnerVolumeSpecName "kube-api-access-6sc5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:54.554914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.554868 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:54.554914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.554902 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sc5c\" (UniqueName: \"kubernetes.io/projected/d8807f9e-1791-40cb-887a-27c3f6c529a7-kube-api-access-6sc5c\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:54.554914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.554912 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8807f9e-1791-40cb-887a-27c3f6c529a7-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:19:54.712725 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.712644 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df505a7-36de-4169-a32b-9fba92b528aa" path="/var/lib/kubelet/pods/1df505a7-36de-4169-a32b-9fba92b528aa/volumes" Nov 03 00:19:54.713029 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:54.713016 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612b0292-9195-4729-bcfb-44ea8d80a9c0" path="/var/lib/kubelet/pods/612b0292-9195-4729-bcfb-44ea8d80a9c0/volumes" Nov 03 00:19:55.180019 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:55.179982 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" event={"ID":"d8807f9e-1791-40cb-887a-27c3f6c529a7","Type":"ContainerDied","Data":"e6f25532b716115b5da50374cd2cc5182ea6b2b7f913a86d5216d7cd79b59ae1"} Nov 03 00:19:55.180259 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:55.180024 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg" Nov 03 00:19:55.180259 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:55.180042 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6f25532b716115b5da50374cd2cc5182ea6b2b7f913a86d5216d7cd79b59ae1" Nov 03 00:19:55.526215 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:19:55.526178 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65"] Nov 03 00:19:56.020650 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:19:56.020611 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:20:00.512612 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:00.512574 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:20:03.761184 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.761129 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm"] Nov 03 00:20:03.761629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.761515 5672 cpu_manager.go:414] "containerMap: removing container" podUID="612b0292-9195-4729-bcfb-44ea8d80a9c0" containerName="payload-processing" Nov 03 00:20:03.761629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.761526 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612b0292-9195-4729-bcfb-44ea8d80a9c0" containerName="payload-processing" Nov 03 00:20:03.761629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.761547 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d8807f9e-1791-40cb-887a-27c3f6c529a7" containerName="revoke-keys" Nov 03 00:20:03.761629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.761553 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8807f9e-1791-40cb-887a-27c3f6c529a7" containerName="revoke-keys" Nov 03 00:20:03.761629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.761610 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="612b0292-9195-4729-bcfb-44ea8d80a9c0" containerName="payload-processing" Nov 03 00:20:03.761629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.761622 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8807f9e-1791-40cb-887a-27c3f6c529a7" containerName="revoke-keys" Nov 03 00:20:03.766116 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.766094 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:03.815435 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.815396 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:20:03.815667 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.815477 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:20:03.815667 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.815500 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:20:03.842166 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.842126 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ba68c1-8fgjm\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:03.842363 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.842300 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ba68c1-8fgjm\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:03.842432 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.842417 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr4lr\" (UniqueName: \"kubernetes.io/projected/3111d9d4-1710-4ddc-aa22-a85a07b60894-kube-api-access-qr4lr\") pod \"maas-api-revoke-sub-42ba68c1-8fgjm\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:03.862295 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.862261 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\"" Nov 03 00:20:03.864615 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.864585 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm"] Nov 03 00:20:03.942969 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.942934 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr4lr\" (UniqueName: \"kubernetes.io/projected/3111d9d4-1710-4ddc-aa22-a85a07b60894-kube-api-access-qr4lr\") pod \"maas-api-revoke-sub-42ba68c1-8fgjm\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:03.943174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.942994 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ba68c1-8fgjm\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:03.943174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.943057 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ba68c1-8fgjm\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:03.943795 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.943765 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ba68c1-8fgjm\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:03.945381 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:03.945360 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ba68c1-8fgjm\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:04.020618 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:04.020537 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr4lr\" (UniqueName: \"kubernetes.io/projected/3111d9d4-1710-4ddc-aa22-a85a07b60894-kube-api-access-qr4lr\") pod \"maas-api-revoke-sub-42ba68c1-8fgjm\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:04.078456 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:04.078409 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:04.086077 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:04.086034 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6"] Nov 03 00:20:04.102591 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:04.102555 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242382cb-cqss6"] Nov 03 00:20:04.448702 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:04.448671 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm"] Nov 03 00:20:04.451066 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:20:04.451032 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3111d9d4_1710_4ddc_aa22_a85a07b60894.slice/crio-67cfe3c4fdd40d0845e24ec939842b170d181ace9c467724f941c276eb468578 WatchSource:0}: Error finding container 67cfe3c4fdd40d0845e24ec939842b170d181ace9c467724f941c276eb468578: Status 404 returned error can't find the container with id 67cfe3c4fdd40d0845e24ec939842b170d181ace9c467724f941c276eb468578 Nov 03 00:20:04.711241 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:04.711134 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2144ab98-3fe5-4849-b80c-507d4e015a46" path="/var/lib/kubelet/pods/2144ab98-3fe5-4849-b80c-507d4e015a46/volumes" Nov 03 00:20:05.254309 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:05.254274 5672 generic.go:356] "Generic (PLEG): container finished" podID="3111d9d4-1710-4ddc-aa22-a85a07b60894" containerID="d8109584b8e21538995aae98bf52a092ef8afd2899bb6441edcdc60dda613481" exitCode=0 Nov 03 00:20:05.254683 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:05.254359 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" event={"ID":"3111d9d4-1710-4ddc-aa22-a85a07b60894","Type":"ContainerDied","Data":"d8109584b8e21538995aae98bf52a092ef8afd2899bb6441edcdc60dda613481"} Nov 03 00:20:05.254683 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:05.254415 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" event={"ID":"3111d9d4-1710-4ddc-aa22-a85a07b60894","Type":"ContainerStarted","Data":"67cfe3c4fdd40d0845e24ec939842b170d181ace9c467724f941c276eb468578"} Nov 03 00:20:06.395542 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:06.395512 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:06.465387 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:06.465353 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3111d9d4-1710-4ddc-aa22-a85a07b60894-kube-api-access-qr4lr\" (UniqueName: \"kubernetes.io/projected/3111d9d4-1710-4ddc-aa22-a85a07b60894-kube-api-access-qr4lr\") pod \"3111d9d4-1710-4ddc-aa22-a85a07b60894\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " Nov 03 00:20:06.465556 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:06.465458 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-xks-ca\") pod \"3111d9d4-1710-4ddc-aa22-a85a07b60894\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " Nov 03 00:20:06.465556 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:06.465511 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-openshift-ca\") pod \"3111d9d4-1710-4ddc-aa22-a85a07b60894\" (UID: \"3111d9d4-1710-4ddc-aa22-a85a07b60894\") " Nov 03 00:20:06.465893 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:06.465867 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-openshift-ca" pod "3111d9d4-1710-4ddc-aa22-a85a07b60894" (UID: "3111d9d4-1710-4ddc-aa22-a85a07b60894"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:20:06.467554 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:06.467531 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-xks-ca" pod "3111d9d4-1710-4ddc-aa22-a85a07b60894" (UID: "3111d9d4-1710-4ddc-aa22-a85a07b60894"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:20:06.467648 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:06.467633 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3111d9d4-1710-4ddc-aa22-a85a07b60894-kube-api-access-qr4lr" pod "3111d9d4-1710-4ddc-aa22-a85a07b60894" (UID: "3111d9d4-1710-4ddc-aa22-a85a07b60894"). InnerVolumeSpecName "kube-api-access-qr4lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:06.566647 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:06.566561 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr4lr\" (UniqueName: \"kubernetes.io/projected/3111d9d4-1710-4ddc-aa22-a85a07b60894-kube-api-access-qr4lr\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:06.566647 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:06.566590 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:06.566647 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:06.566602 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3111d9d4-1710-4ddc-aa22-a85a07b60894-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:07.267658 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:07.267626 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" Nov 03 00:20:07.267907 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:07.267627 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm" event={"ID":"3111d9d4-1710-4ddc-aa22-a85a07b60894","Type":"ContainerDied","Data":"67cfe3c4fdd40d0845e24ec939842b170d181ace9c467724f941c276eb468578"} Nov 03 00:20:07.267907 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:07.267756 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67cfe3c4fdd40d0845e24ec939842b170d181ace9c467724f941c276eb468578" Nov 03 00:20:10.546416 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.546373 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" containerID="cri-o://f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620" gracePeriod=45 Nov 03 00:20:10.807640 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.807575 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 03 00:20:10.902598 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.902560 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97473a88-1d49-4873-970e-914e29a6908f-tls-certs\" (UniqueName: \"kubernetes.io/secret/97473a88-1d49-4873-970e-914e29a6908f-tls-certs\") pod \"97473a88-1d49-4873-970e-914e29a6908f\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " Nov 03 00:20:10.902598 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.902603 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-model-cache\") pod \"97473a88-1d49-4873-970e-914e29a6908f\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " Nov 03 00:20:10.902851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.902644 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-tmp-dir\") pod \"97473a88-1d49-4873-970e-914e29a6908f\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " Nov 03 00:20:10.902851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.902676 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-dshm\") pod \"97473a88-1d49-4873-970e-914e29a6908f\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " Nov 03 00:20:10.902851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.902733 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97473a88-1d49-4873-970e-914e29a6908f-kube-api-access-rkwz2\" (UniqueName: \"kubernetes.io/projected/97473a88-1d49-4873-970e-914e29a6908f-kube-api-access-rkwz2\") pod \"97473a88-1d49-4873-970e-914e29a6908f\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " Nov 03 00:20:10.902851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.902790 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-home\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-home\") pod \"97473a88-1d49-4873-970e-914e29a6908f\" (UID: \"97473a88-1d49-4873-970e-914e29a6908f\") " Nov 03 00:20:10.903071 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.902919 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-tmp-dir" pod "97473a88-1d49-4873-970e-914e29a6908f" (UID: "97473a88-1d49-4873-970e-914e29a6908f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:10.903071 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.902947 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-model-cache" pod "97473a88-1d49-4873-970e-914e29a6908f" (UID: "97473a88-1d49-4873-970e-914e29a6908f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:10.903192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.903111 5672 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-model-cache\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:10.903192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.903070 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-home" pod "97473a88-1d49-4873-970e-914e29a6908f" (UID: "97473a88-1d49-4873-970e-914e29a6908f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:10.903192 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.903130 5672 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-tmp-dir\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:10.904763 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.904737 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97473a88-1d49-4873-970e-914e29a6908f-tls-certs" pod "97473a88-1d49-4873-970e-914e29a6908f" (UID: "97473a88-1d49-4873-970e-914e29a6908f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:20:10.904893 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.904802 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97473a88-1d49-4873-970e-914e29a6908f-kube-api-access-rkwz2" pod "97473a88-1d49-4873-970e-914e29a6908f" (UID: "97473a88-1d49-4873-970e-914e29a6908f"). InnerVolumeSpecName "kube-api-access-rkwz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:10.904893 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:10.904841 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-dshm" pod "97473a88-1d49-4873-970e-914e29a6908f" (UID: "97473a88-1d49-4873-970e-914e29a6908f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:11.004497 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.004458 5672 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97473a88-1d49-4873-970e-914e29a6908f-tls-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:11.004497 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.004485 5672 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-dshm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:11.004497 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.004495 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkwz2\" (UniqueName: \"kubernetes.io/projected/97473a88-1d49-4873-970e-914e29a6908f-kube-api-access-rkwz2\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:11.004497 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.004504 5672 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97473a88-1d49-4873-970e-914e29a6908f-home\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:11.296012 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.295975 5672 generic.go:356] "Generic (PLEG): container finished" podID="97473a88-1d49-4873-970e-914e29a6908f" containerID="f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620" exitCode=0 Nov 03 00:20:11.296237 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.296022 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" Nov 03 00:20:11.296237 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.296061 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" event={"ID":"97473a88-1d49-4873-970e-914e29a6908f","Type":"ContainerDied","Data":"f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620"} Nov 03 00:20:11.296237 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.296108 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65" event={"ID":"97473a88-1d49-4873-970e-914e29a6908f","Type":"ContainerDied","Data":"bdb3070af0b31ecb473259c85eba1d36021de5588870944dccd108e2f616b870"} Nov 03 00:20:11.296237 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.296126 5672 scope.go:122] "RemoveContainer" containerID="f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620" Nov 03 00:20:11.305530 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.305511 5672 scope.go:122] "RemoveContainer" containerID="f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620" Nov 03 00:20:11.305788 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:20:11.305769 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620\": container with ID starting with f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620 not found: ID does not exist" containerID="f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620" Nov 03 00:20:11.305862 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.305802 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620"} err="failed to get container status \"f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620\": rpc error: code = NotFound desc = could not find container \"f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620\": container with ID starting with f61dc6d52218aac421abc8b98d3efa8f9cfa9817b222f4cd761effd35e93a620 not found: ID does not exist" Nov 03 00:20:11.453264 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.453230 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65"] Nov 03 00:20:11.459471 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:11.459440 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/distinct-w4-fb6e1d-kserve-5666945557-h6f65"] Nov 03 00:20:12.711587 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:12.711556 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97473a88-1d49-4873-970e-914e29a6908f" path="/var/lib/kubelet/pods/97473a88-1d49-4873-970e-914e29a6908f/volumes" Nov 03 00:20:23.048784 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:23.048743 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:20:24.902242 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:24.902201 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv"] Nov 03 00:20:24.902697 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:24.902626 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3111d9d4-1710-4ddc-aa22-a85a07b60894" containerName="revoke-keys" Nov 03 00:20:24.902697 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:24.902638 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3111d9d4-1710-4ddc-aa22-a85a07b60894" containerName="revoke-keys" Nov 03 00:20:24.902697 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:24.902657 5672 cpu_manager.go:414] "containerMap: removing container" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" Nov 03 00:20:24.902697 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:24.902662 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" Nov 03 00:20:24.902851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:24.902721 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="97473a88-1d49-4873-970e-914e29a6908f" containerName="main" Nov 03 00:20:24.902851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:24.902733 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3111d9d4-1710-4ddc-aa22-a85a07b60894" containerName="revoke-keys" Nov 03 00:20:24.907491 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:24.907470 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:24.956393 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:24.956366 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fb6e1d-openshift-default-dockercfg-5lz5h\"" Nov 03 00:20:25.033891 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.033857 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.033891 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.033895 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.034120 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.033914 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.034120 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.034055 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.034120 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.034090 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.034254 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.034180 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.034254 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.034236 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.034328 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.034316 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xxtm\" (UniqueName: \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-kube-api-access-2xxtm\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.034371 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.034346 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.096627 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.096586 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv"] Nov 03 00:20:25.134925 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.134887 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135107 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.134930 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135107 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135060 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135220 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135141 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135220 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135187 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135220 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135211 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135374 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135273 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135374 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135323 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xxtm\" (UniqueName: \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-kube-api-access-2xxtm\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135374 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135360 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135539 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135378 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135736 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135706 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135736 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135720 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135736 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135731 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.135969 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.135917 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.137339 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.137315 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.137781 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.137758 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.188368 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.188292 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.188368 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.188343 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xxtm\" (UniqueName: \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-kube-api-access-2xxtm\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.220743 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.220701 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:25.634203 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.634174 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv"] Nov 03 00:20:25.635703 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:20:25.635669 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff2ffe9_d9b4_4ece_90f0_516b96bbe674.slice/crio-609ace28bdc5162abd2ca5062bfabc070811d6705d91648b3cda9542f976e237 WatchSource:0}: Error finding container 609ace28bdc5162abd2ca5062bfabc070811d6705d91648b3cda9542f976e237: Status 404 returned error can't find the container with id 609ace28bdc5162abd2ca5062bfabc070811d6705d91648b3cda9542f976e237 Nov 03 00:20:25.638042 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.638008 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:20:25.638181 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.638092 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:20:25.638181 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:25.638134 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:20:26.026209 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:20:26.026159 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:20:26.382468 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:26.382372 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" event={"ID":"cff2ffe9-d9b4-4ece-90f0-516b96bbe674","Type":"ContainerStarted","Data":"20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2"} Nov 03 00:20:26.382468 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:26.382440 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" event={"ID":"cff2ffe9-d9b4-4ece-90f0-516b96bbe674","Type":"ContainerStarted","Data":"609ace28bdc5162abd2ca5062bfabc070811d6705d91648b3cda9542f976e237"} Nov 03 00:20:26.536415 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:26.536340 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" podStartSLOduration=2.53631896 podStartE2EDuration="2.53631896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:26.534844498 +0000 UTC m=+5219.942156330" watchObservedRunningTime="2026-11-03 00:20:26.53631896 +0000 UTC m=+5219.943630789" Nov 03 00:20:27.221908 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:27.221869 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:27.223391 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:27.223367 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 00:20:27.223476 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:27.223416 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" podUID="cff2ffe9-d9b4-4ece-90f0-516b96bbe674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 00:20:28.221680 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:28.221646 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 00:20:28.221892 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:28.221701 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" podUID="cff2ffe9-d9b4-4ece-90f0-516b96bbe674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 00:20:29.221775 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:29.221737 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 00:20:29.222200 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:29.221794 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" podUID="cff2ffe9-d9b4-4ece-90f0-516b96bbe674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 00:20:30.221345 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:30.221306 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 00:20:30.221517 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:30.221373 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" podUID="cff2ffe9-d9b4-4ece-90f0-516b96bbe674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 00:20:31.221904 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:31.221863 5672 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 00:20:31.222363 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:31.221923 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" podUID="cff2ffe9-d9b4-4ece-90f0-516b96bbe674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 00:20:32.225982 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:32.225950 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:32.226588 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:32.226364 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:32.227225 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:32.227206 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:35.511876 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:35.511836 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv"] Nov 03 00:20:35.512395 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:35.512036 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" podUID="cff2ffe9-d9b4-4ece-90f0-516b96bbe674" containerName="istio-proxy" containerID="cri-o://20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2" gracePeriod=30 Nov 03 00:20:37.644624 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:37.644589 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-kbd6z_2c50862d-0486-4a19-9f75-4bf4f6d27dc2/kuadrant-console-plugin/0.log" Nov 03 00:20:38.402770 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:38.402734 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rqrwv_027227b3-6a47-4b4a-8506-c8bf8763f4d3/manager/0.log" Nov 03 00:20:38.644611 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:38.644579 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6b47m_8e9db611-6d4f-4748-a3f4-4e8cfff61634/limitador/0.log" Nov 03 00:20:40.088529 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.088494 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2f5hv_c1428a43-3796-41a6-b8e1-79c8062885b3/manager/0.log" Nov 03 00:20:40.579431 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.579398 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-gt9zc_db028e9f-d723-4a0c-998e-e1712a86b183/manager/0.log" Nov 03 00:20:40.754824 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.754794 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:40.888994 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.888868 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-kube-api-access-2xxtm\" (UniqueName: \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-kube-api-access-2xxtm\") pod \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " Nov 03 00:20:40.888994 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.888966 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-data\") pod \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " Nov 03 00:20:40.889418 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889001 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-envoy\") pod \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " Nov 03 00:20:40.889418 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889040 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-token\" (UniqueName: \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-token\") pod \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " Nov 03 00:20:40.889418 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889074 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istiod-ca-cert\") pod \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " Nov 03 00:20:40.889418 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889105 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-certs\") pod \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " Nov 03 00:20:40.889418 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889136 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-podinfo\") pod \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " Nov 03 00:20:40.889418 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889187 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-socket\") pod \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " Nov 03 00:20:40.889418 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889230 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-credential-socket\") pod \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\" (UID: \"cff2ffe9-d9b4-4ece-90f0-516b96bbe674\") " Nov 03 00:20:40.889418 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889402 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-data" pod "cff2ffe9-d9b4-4ece-90f0-516b96bbe674" (UID: "cff2ffe9-d9b4-4ece-90f0-516b96bbe674"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.890110 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889593 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-certs" pod "cff2ffe9-d9b4-4ece-90f0-516b96bbe674" (UID: "cff2ffe9-d9b4-4ece-90f0-516b96bbe674"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.890110 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889639 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-credential-socket" pod "cff2ffe9-d9b4-4ece-90f0-516b96bbe674" (UID: "cff2ffe9-d9b4-4ece-90f0-516b96bbe674"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.890110 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889709 5672 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-data\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.890110 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889734 5672 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.890110 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.889923 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istiod-ca-cert" pod "cff2ffe9-d9b4-4ece-90f0-516b96bbe674" (UID: "cff2ffe9-d9b4-4ece-90f0-516b96bbe674"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:20:40.890464 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.890131 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-socket" pod "cff2ffe9-d9b4-4ece-90f0-516b96bbe674" (UID: "cff2ffe9-d9b4-4ece-90f0-516b96bbe674"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.893628 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.893512 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-envoy" pod "cff2ffe9-d9b4-4ece-90f0-516b96bbe674" (UID: "cff2ffe9-d9b4-4ece-90f0-516b96bbe674"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:20:40.893768 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.893650 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-podinfo" pod "cff2ffe9-d9b4-4ece-90f0-516b96bbe674" (UID: "cff2ffe9-d9b4-4ece-90f0-516b96bbe674"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:20:40.893768 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.893686 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-kube-api-access-2xxtm" pod "cff2ffe9-d9b4-4ece-90f0-516b96bbe674" (UID: "cff2ffe9-d9b4-4ece-90f0-516b96bbe674"). InnerVolumeSpecName "kube-api-access-2xxtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:40.893917 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.893792 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-token" pod "cff2ffe9-d9b4-4ece-90f0-516b96bbe674" (UID: "cff2ffe9-d9b4-4ece-90f0-516b96bbe674"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:40.990249 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.990214 5672 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-envoy\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.990249 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.990250 5672 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-token\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.990249 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.990266 5672 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istiod-ca-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.990610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.990282 5672 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-istio-podinfo\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.990610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.990296 5672 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-workload-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.990610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.990309 5672 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-credential-socket\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:40.990610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:40.990323 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xxtm\" (UniqueName: \"kubernetes.io/projected/cff2ffe9-d9b4-4ece-90f0-516b96bbe674-kube-api-access-2xxtm\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:20:41.221056 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.221019 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b67978596-nrwt6_906286bb-6770-4daa-aa76-514a51f1347e/manager/0.log" Nov 03 00:20:41.429765 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.429729 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fg848_30e01866-fe73-471f-8e2c-f126cb04b8b8/server/0.log" Nov 03 00:20:41.470734 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.470698 5672 generic.go:356] "Generic (PLEG): container finished" podID="cff2ffe9-d9b4-4ece-90f0-516b96bbe674" containerID="20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2" exitCode=0 Nov 03 00:20:41.470910 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.470769 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" Nov 03 00:20:41.470910 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.470783 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" event={"ID":"cff2ffe9-d9b4-4ece-90f0-516b96bbe674","Type":"ContainerDied","Data":"20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2"} Nov 03 00:20:41.470910 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.470866 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv" event={"ID":"cff2ffe9-d9b4-4ece-90f0-516b96bbe674","Type":"ContainerDied","Data":"609ace28bdc5162abd2ca5062bfabc070811d6705d91648b3cda9542f976e237"} Nov 03 00:20:41.470910 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.470888 5672 scope.go:122] "RemoveContainer" containerID="20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2" Nov 03 00:20:41.480774 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.480745 5672 scope.go:122] "RemoveContainer" containerID="20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2" Nov 03 00:20:41.481058 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:20:41.481035 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2\": container with ID starting with 20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2 not found: ID does not exist" containerID="20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2" Nov 03 00:20:41.481115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.481067 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2"} err="failed to get container status \"20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2\": rpc error: code = NotFound desc = could not find container \"20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2\": container with ID starting with 20dc7665537d75ab6667c175bf52417b7001bb31668d63f64ae4f4b31fa109d2 not found: ID does not exist" Nov 03 00:20:41.674624 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.674589 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv"] Nov 03 00:20:41.680001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.679967 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-d46vv"] Nov 03 00:20:41.928018 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:41.927926 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-knpb6_eeb11765-1103-4234-aeef-55ecae7338dd/proxy/0.log" Nov 03 00:20:42.045923 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:42.045888 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc"] Nov 03 00:20:42.048535 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:42.048505 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820b1c31-b6rwc"] Nov 03 00:20:42.711052 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:42.711021 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662d88bc-eebb-4c34-8f3a-9f36ea9d1002" path="/var/lib/kubelet/pods/662d88bc-eebb-4c34-8f3a-9f36ea9d1002/volumes" Nov 03 00:20:42.711483 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:42.711466 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff2ffe9-d9b4-4ece-90f0-516b96bbe674" path="/var/lib/kubelet/pods/cff2ffe9-d9b4-4ece-90f0-516b96bbe674/volumes" Nov 03 00:20:52.044259 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:52.044219 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh"] Nov 03 00:20:52.090688 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:52.090654 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b27326ad-6szrh"] Nov 03 00:20:52.711483 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:20:52.711453 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19958d33-9ec4-44ca-98b7-913dc66aecef" path="/var/lib/kubelet/pods/19958d33-9ec4-44ca-98b7-913dc66aecef/volumes" Nov 03 00:20:56.020193 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:20:56.020134 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:21:04.043796 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:04.043760 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd"] Nov 03 00:21:04.090008 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:04.089983 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a62ed18-nzdqd"] Nov 03 00:21:04.711309 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:04.711272 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d39c8a-29cd-4a9e-8e70-5e7c299b7892" path="/var/lib/kubelet/pods/09d39c8a-29cd-4a9e-8e70-5e7c299b7892/volumes" Nov 03 00:21:05.058097 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:05.058059 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:21:15.041808 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:15.041770 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt"] Nov 03 00:21:15.089274 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:15.089242 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f78cd7-lb7tt"] Nov 03 00:21:16.711568 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:16.711534 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7646e0-c37d-4cef-bc05-55e3c2e9f891" path="/var/lib/kubelet/pods/0b7646e0-c37d-4cef-bc05-55e3c2e9f891/volumes" Nov 03 00:21:17.247966 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.247929 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw"] Nov 03 00:21:17.248313 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.248298 5672 cpu_manager.go:414] "containerMap: removing container" podUID="cff2ffe9-d9b4-4ece-90f0-516b96bbe674" containerName="istio-proxy" Nov 03 00:21:17.248384 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.248313 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff2ffe9-d9b4-4ece-90f0-516b96bbe674" containerName="istio-proxy" Nov 03 00:21:17.248423 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.248395 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff2ffe9-d9b4-4ece-90f0-516b96bbe674" containerName="istio-proxy" Nov 03 00:21:17.251748 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.251726 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:17.313402 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.313373 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:17.313559 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.313406 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9tf\" (UniqueName: \"kubernetes.io/projected/15950b4c-0e99-49b3-886e-706992ebb958-kube-api-access-7l9tf\") pod \"maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:17.313559 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.313445 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:17.347708 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.347680 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:21:17.347708 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.347687 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:21:17.347893 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.347756 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:21:17.347893 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.347816 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:21:17.350130 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.350109 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw"] Nov 03 00:21:17.414780 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.414755 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:17.414951 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.414787 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9tf\" (UniqueName: \"kubernetes.io/projected/15950b4c-0e99-49b3-886e-706992ebb958-kube-api-access-7l9tf\") pod \"maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:17.414951 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.414825 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:17.415547 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.415523 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:17.417128 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.417105 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:17.468094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.468056 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9tf\" (UniqueName: \"kubernetes.io/projected/15950b4c-0e99-49b3-886e-706992ebb958-kube-api-access-7l9tf\") pod \"maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:17.564292 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.564209 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:17.931557 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:17.931531 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw"] Nov 03 00:21:17.933357 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:21:17.933330 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15950b4c_0e99_49b3_886e_706992ebb958.slice/crio-97524ae0f0d9aff0c2776fd4d5dbbddaa2a538c16e146576d253ea2a19c862e0 WatchSource:0}: Error finding container 97524ae0f0d9aff0c2776fd4d5dbbddaa2a538c16e146576d253ea2a19c862e0: Status 404 returned error can't find the container with id 97524ae0f0d9aff0c2776fd4d5dbbddaa2a538c16e146576d253ea2a19c862e0 Nov 03 00:21:18.682836 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:18.682738 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" event={"ID":"15950b4c-0e99-49b3-886e-706992ebb958","Type":"ContainerStarted","Data":"9d4010a5aba2cffe8db064c0139c99c1bd3c776750cd10238342d9986f81c32d"} Nov 03 00:21:18.682836 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:18.682800 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" event={"ID":"15950b4c-0e99-49b3-886e-706992ebb958","Type":"ContainerStarted","Data":"97524ae0f0d9aff0c2776fd4d5dbbddaa2a538c16e146576d253ea2a19c862e0"} Nov 03 00:21:18.791342 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:18.791292 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" podStartSLOduration=1.791278201 podStartE2EDuration="1.791278201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:18.789347961 +0000 UTC m=+5272.196659791" watchObservedRunningTime="2026-11-03 00:21:18.791278201 +0000 UTC m=+5272.198590031" Nov 03 00:21:19.690160 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:19.690117 5672 generic.go:356] "Generic (PLEG): container finished" podID="15950b4c-0e99-49b3-886e-706992ebb958" containerID="9d4010a5aba2cffe8db064c0139c99c1bd3c776750cd10238342d9986f81c32d" exitCode=0 Nov 03 00:21:19.690551 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:19.690182 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" event={"ID":"15950b4c-0e99-49b3-886e-706992ebb958","Type":"ContainerDied","Data":"9d4010a5aba2cffe8db064c0139c99c1bd3c776750cd10238342d9986f81c32d"} Nov 03 00:21:20.827876 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:20.827845 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:20.947080 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:20.947002 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-openshift-ca\") pod \"15950b4c-0e99-49b3-886e-706992ebb958\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " Nov 03 00:21:20.947080 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:20.947050 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-xks-ca\") pod \"15950b4c-0e99-49b3-886e-706992ebb958\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " Nov 03 00:21:20.947080 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:20.947072 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15950b4c-0e99-49b3-886e-706992ebb958-kube-api-access-7l9tf\" (UniqueName: \"kubernetes.io/projected/15950b4c-0e99-49b3-886e-706992ebb958-kube-api-access-7l9tf\") pod \"15950b4c-0e99-49b3-886e-706992ebb958\" (UID: \"15950b4c-0e99-49b3-886e-706992ebb958\") " Nov 03 00:21:20.947447 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:20.947417 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-openshift-ca" pod "15950b4c-0e99-49b3-886e-706992ebb958" (UID: "15950b4c-0e99-49b3-886e-706992ebb958"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:21:20.949141 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:20.949117 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-xks-ca" pod "15950b4c-0e99-49b3-886e-706992ebb958" (UID: "15950b4c-0e99-49b3-886e-706992ebb958"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:21:20.949309 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:20.949286 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15950b4c-0e99-49b3-886e-706992ebb958-kube-api-access-7l9tf" pod "15950b4c-0e99-49b3-886e-706992ebb958" (UID: "15950b4c-0e99-49b3-886e-706992ebb958"). InnerVolumeSpecName "kube-api-access-7l9tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:21.048370 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:21.048344 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:21.048370 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:21.048369 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15950b4c-0e99-49b3-886e-706992ebb958-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:21.048518 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:21.048379 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l9tf\" (UniqueName: \"kubernetes.io/projected/15950b4c-0e99-49b3-886e-706992ebb958-kube-api-access-7l9tf\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:21.702593 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:21.702561 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" Nov 03 00:21:21.702776 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:21.702595 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw" event={"ID":"15950b4c-0e99-49b3-886e-706992ebb958","Type":"ContainerDied","Data":"97524ae0f0d9aff0c2776fd4d5dbbddaa2a538c16e146576d253ea2a19c862e0"} Nov 03 00:21:21.702776 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:21.702648 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97524ae0f0d9aff0c2776fd4d5dbbddaa2a538c16e146576d253ea2a19c862e0" Nov 03 00:21:26.019330 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:21:26.019296 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:21:30.163841 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:30.163808 5672 scope.go:122] "RemoveContainer" containerID="041e26711ebfc181ad32b0dcbd7388737d62638b01904e79f580f3d015c1c34c" Nov 03 00:21:30.172858 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:30.172839 5672 scope.go:122] "RemoveContainer" containerID="867ca72a64b95bf3a2e18161a648f1f76a93fef47986594d79618faace5be2cf" Nov 03 00:21:36.715407 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:36.715373 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw"] Nov 03 00:21:36.761712 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:36.761681 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2a3480b-sl9lw"] Nov 03 00:21:38.043654 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:38.043622 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h"] Nov 03 00:21:38.089918 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:38.089886 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de6af444-qmj4h"] Nov 03 00:21:38.710966 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:38.710931 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15950b4c-0e99-49b3-886e-706992ebb958" path="/var/lib/kubelet/pods/15950b4c-0e99-49b3-886e-706992ebb958/volumes" Nov 03 00:21:38.711350 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:38.711336 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424d3364-432e-4dde-ac07-75108c4ca592" path="/var/lib/kubelet/pods/424d3364-432e-4dde-ac07-75108c4ca592/volumes" Nov 03 00:21:41.207034 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:41.206995 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:21:42.402573 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.402538 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b67978596-nrwt6"] Nov 03 00:21:42.403086 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.402784 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-b67978596-nrwt6" podUID="906286bb-6770-4daa-aa76-514a51f1347e" containerName="manager" containerID="cri-o://737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d" gracePeriod=30 Nov 03 00:21:42.652442 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.652411 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 03 00:21:42.740815 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.740783 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906286bb-6770-4daa-aa76-514a51f1347e-kube-api-access-d5qp6\" (UniqueName: \"kubernetes.io/projected/906286bb-6770-4daa-aa76-514a51f1347e-kube-api-access-d5qp6\") pod \"906286bb-6770-4daa-aa76-514a51f1347e\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " Nov 03 00:21:42.740968 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.740823 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-metrics-certs\") pod \"906286bb-6770-4daa-aa76-514a51f1347e\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " Nov 03 00:21:42.740968 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.740863 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-webhook-cert\") pod \"906286bb-6770-4daa-aa76-514a51f1347e\" (UID: \"906286bb-6770-4daa-aa76-514a51f1347e\") " Nov 03 00:21:42.742922 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.742892 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-webhook-cert" pod "906286bb-6770-4daa-aa76-514a51f1347e" (UID: "906286bb-6770-4daa-aa76-514a51f1347e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:21:42.743065 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.742921 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-metrics-certs" pod "906286bb-6770-4daa-aa76-514a51f1347e" (UID: "906286bb-6770-4daa-aa76-514a51f1347e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:21:42.743065 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.742952 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906286bb-6770-4daa-aa76-514a51f1347e-kube-api-access-d5qp6" pod "906286bb-6770-4daa-aa76-514a51f1347e" (UID: "906286bb-6770-4daa-aa76-514a51f1347e"). InnerVolumeSpecName "kube-api-access-d5qp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:42.818396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.818362 5672 generic.go:356] "Generic (PLEG): container finished" podID="906286bb-6770-4daa-aa76-514a51f1347e" containerID="737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d" exitCode=0 Nov 03 00:21:42.818585 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.818432 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b67978596-nrwt6" Nov 03 00:21:42.818585 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.818442 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b67978596-nrwt6" event={"ID":"906286bb-6770-4daa-aa76-514a51f1347e","Type":"ContainerDied","Data":"737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d"} Nov 03 00:21:42.818585 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.818500 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b67978596-nrwt6" event={"ID":"906286bb-6770-4daa-aa76-514a51f1347e","Type":"ContainerDied","Data":"d0f21fb3d134dfd8092452ee1b94e370bfcc4e30ffa46443e1b393784f677fc8"} Nov 03 00:21:42.818585 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.818512 5672 scope.go:122] "RemoveContainer" containerID="737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d" Nov 03 00:21:42.828974 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.828952 5672 scope.go:122] "RemoveContainer" containerID="737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d" Nov 03 00:21:42.829276 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:21:42.829256 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d\": container with ID starting with 737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d not found: ID does not exist" containerID="737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d" Nov 03 00:21:42.829337 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.829286 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d"} err="failed to get container status \"737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d\": rpc error: code = NotFound desc = could not find container \"737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d\": container with ID starting with 737b5c93f1116f8994c56e691bc412e4be0d9901830f552e97891b7a021b950d not found: ID does not exist" Nov 03 00:21:42.841684 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.841663 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5qp6\" (UniqueName: \"kubernetes.io/projected/906286bb-6770-4daa-aa76-514a51f1347e-kube-api-access-d5qp6\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:42.841784 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.841686 5672 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-metrics-certs\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:42.841784 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.841696 5672 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/906286bb-6770-4daa-aa76-514a51f1347e-webhook-cert\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:21:42.972002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.971967 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b67978596-nrwt6"] Nov 03 00:21:42.973917 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:42.973894 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b67978596-nrwt6"] Nov 03 00:21:44.711027 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:44.710993 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="906286bb-6770-4daa-aa76-514a51f1347e" path="/var/lib/kubelet/pods/906286bb-6770-4daa-aa76-514a51f1347e/volumes" Nov 03 00:21:49.043646 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:49.043595 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9"] Nov 03 00:21:49.090791 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:49.090753 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ea58fc-v46n9"] Nov 03 00:21:50.712162 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:50.712120 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe00cb2-b676-42c1-beaf-bce901ce8394" path="/var/lib/kubelet/pods/fbe00cb2-b676-42c1-beaf-bce901ce8394/volumes" Nov 03 00:21:54.044877 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:54.044838 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n"] Nov 03 00:21:54.091206 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:54.091172 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da84f27e-wwn8n"] Nov 03 00:21:54.710975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:21:54.710943 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7f2b93-fd95-4de4-ba73-7971096e0971" path="/var/lib/kubelet/pods/bf7f2b93-fd95-4de4-ba73-7971096e0971/volumes" Nov 03 00:21:56.019465 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:21:56.019429 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:22:07.043635 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:07.043549 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt"] Nov 03 00:22:07.089621 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:07.089582 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49634525-59xtt"] Nov 03 00:22:08.710801 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:08.710768 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a242d460-6053-4ead-8930-bc07066a2e97" path="/var/lib/kubelet/pods/a242d460-6053-4ead-8930-bc07066a2e97/volumes" Nov 03 00:22:21.047415 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:21.047379 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x"] Nov 03 00:22:21.093413 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:21.093386 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cefa818-6687x"] Nov 03 00:22:22.710284 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:22.710252 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20c3945-6e41-4ad7-96a6-baf1173a172f" path="/var/lib/kubelet/pods/b20c3945-6e41-4ad7-96a6-baf1173a172f/volumes" Nov 03 00:22:26.019056 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:22:26.019022 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:22:30.202779 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:30.202746 5672 scope.go:122] "RemoveContainer" containerID="99947a9f759fdabcaa95794fd4d150a9eba322c778760503acaea9a27b113197" Nov 03 00:22:30.211274 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:30.211245 5672 scope.go:122] "RemoveContainer" containerID="4f7ddc338d904577a98bc4d61c3f5ebd1c697bafc98db37d91e409ce94369c66" Nov 03 00:22:30.219284 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:30.219261 5672 scope.go:122] "RemoveContainer" containerID="c2745dbe2b7060769a077d5eaaaeb1daa7cc3f0744a2862d0740cce7be3ac021" Nov 03 00:22:30.227336 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:30.227316 5672 scope.go:122] "RemoveContainer" containerID="8d0566ac2a4e1c065f81c3436fd13d06852c385ff5288d552398267079e8a973" Nov 03 00:22:32.043300 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:32.043268 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z"] Nov 03 00:22:32.089183 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:32.089131 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c35399aa-lj57z"] Nov 03 00:22:32.710907 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:32.710875 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8991896-637e-4812-a82b-d2e9175f1567" path="/var/lib/kubelet/pods/b8991896-637e-4812-a82b-d2e9175f1567/volumes" Nov 03 00:22:43.042298 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:43.042262 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg"] Nov 03 00:22:43.088812 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:43.088783 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f92101-vgnsg"] Nov 03 00:22:44.711311 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:44.711278 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7ba1f4-fbe0-4c8e-b334-3959010b1fb9" path="/var/lib/kubelet/pods/de7ba1f4-fbe0-4c8e-b334-3959010b1fb9/volumes" Nov 03 00:22:47.230060 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:47.230027 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:22:54.042719 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:54.042681 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r"] Nov 03 00:22:54.089415 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:54.089380 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f5b73a-fvv2r"] Nov 03 00:22:54.711278 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:22:54.711243 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39c9d5b-887e-4ec0-ae61-011842d2af4c" path="/var/lib/kubelet/pods/b39c9d5b-887e-4ec0-ae61-011842d2af4c/volumes" Nov 03 00:22:56.019940 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:22:56.019903 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:23:09.042417 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:09.042379 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd"] Nov 03 00:23:09.088823 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:09.088777 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b99f726d-tkjkd"] Nov 03 00:23:09.245032 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:09.245002 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:23:10.710947 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:10.710914 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f456c7bf-4939-4c59-afd0-d7e8ad0762e8" path="/var/lib/kubelet/pods/f456c7bf-4939-4c59-afd0-d7e8ad0762e8/volumes" Nov 03 00:23:18.902317 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:18.902274 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv"] Nov 03 00:23:18.902832 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:18.902815 5672 cpu_manager.go:414] "containerMap: removing container" podUID="906286bb-6770-4daa-aa76-514a51f1347e" containerName="manager" Nov 03 00:23:18.902877 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:18.902832 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906286bb-6770-4daa-aa76-514a51f1347e" containerName="manager" Nov 03 00:23:18.902877 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:18.902850 5672 cpu_manager.go:414] "containerMap: removing container" podUID="15950b4c-0e99-49b3-886e-706992ebb958" containerName="revoke-keys" Nov 03 00:23:18.902877 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:18.902858 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15950b4c-0e99-49b3-886e-706992ebb958" containerName="revoke-keys" Nov 03 00:23:18.902977 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:18.902956 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="906286bb-6770-4daa-aa76-514a51f1347e" containerName="manager" Nov 03 00:23:18.902977 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:18.902973 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="15950b4c-0e99-49b3-886e-706992ebb958" containerName="revoke-keys" Nov 03 00:23:18.905931 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:18.905907 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.083550 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.083497 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.083752 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.083556 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.083752 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.083660 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.083752 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.083737 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.083932 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.083800 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.083932 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.083841 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzlpr\" (UniqueName: \"kubernetes.io/projected/26ee8627-3e4b-486b-9e59-f646fa5f5d65-kube-api-access-rzlpr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.084036 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.083932 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.084036 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.083973 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.084110 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.084068 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.094667 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.094639 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv"] Nov 03 00:23:19.184764 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.184663 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.184764 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.184715 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.184764 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.184748 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.185065 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.184806 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.185065 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.184843 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.185065 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.184875 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzlpr\" (UniqueName: \"kubernetes.io/projected/26ee8627-3e4b-486b-9e59-f646fa5f5d65-kube-api-access-rzlpr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.185065 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.184917 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.185065 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.184953 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.185065 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.184986 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.185544 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.185435 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.185611 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.185557 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.185719 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.185690 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.185796 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.185721 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.185991 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.185966 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.188049 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.188021 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.188197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.188181 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.240835 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.240800 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzlpr\" (UniqueName: \"kubernetes.io/projected/26ee8627-3e4b-486b-9e59-f646fa5f5d65-kube-api-access-rzlpr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.241300 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.241276 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ee8627-3e4b-486b-9e59-f646fa5f5d65-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-gt4gv\" (UID: \"26ee8627-3e4b-486b-9e59-f646fa5f5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.522935 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.522896 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:19.960482 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.960456 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv"] Nov 03 00:23:19.962640 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:23:19.962611 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ee8627_3e4b_486b_9e59_f646fa5f5d65.slice/crio-07b800c2a6dcc212e2aa58ebbadadf0819d3d32d54bb869fb9a3a5bb39cfd4b5 WatchSource:0}: Error finding container 07b800c2a6dcc212e2aa58ebbadadf0819d3d32d54bb869fb9a3a5bb39cfd4b5: Status 404 returned error can't find the container with id 07b800c2a6dcc212e2aa58ebbadadf0819d3d32d54bb869fb9a3a5bb39cfd4b5 Nov 03 00:23:19.964903 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.964868 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:19.964994 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.964943 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:19.964994 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:19.964972 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:20.345449 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:20.345407 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" event={"ID":"26ee8627-3e4b-486b-9e59-f646fa5f5d65","Type":"ContainerStarted","Data":"732268897cc81ce08a8cff0ca4013f3db66f86775516b1a09600c6da1b93014e"} Nov 03 00:23:20.345693 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:20.345475 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" event={"ID":"26ee8627-3e4b-486b-9e59-f646fa5f5d65","Type":"ContainerStarted","Data":"07b800c2a6dcc212e2aa58ebbadadf0819d3d32d54bb869fb9a3a5bb39cfd4b5"} Nov 03 00:23:20.456786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:20.456717 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" podStartSLOduration=2.45669869 podStartE2EDuration="2.45669869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:20.45611809 +0000 UTC m=+5393.863429921" watchObservedRunningTime="2026-11-03 00:23:20.45669869 +0000 UTC m=+5393.864010521" Nov 03 00:23:20.523716 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:20.523686 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:21.528350 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:21.528315 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 00:23:21.529173 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:21.528899 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" podUID="26ee8627-3e4b-486b-9e59-f646fa5f5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 00:23:22.523426 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:22.523377 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 00:23:22.523644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:22.523451 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" podUID="26ee8627-3e4b-486b-9e59-f646fa5f5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 00:23:23.414557 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:23.414519 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:23:23.523441 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:23.523401 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 00:23:23.523631 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:23.523470 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" podUID="26ee8627-3e4b-486b-9e59-f646fa5f5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 00:23:24.523491 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:24.523451 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 00:23:24.523920 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:24.523510 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" podUID="26ee8627-3e4b-486b-9e59-f646fa5f5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 00:23:25.528174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:25.524310 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 00:23:25.528174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:25.524376 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" podUID="26ee8627-3e4b-486b-9e59-f646fa5f5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 00:23:26.020648 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:23:26.020605 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:23:26.043437 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:26.043405 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j"] Nov 03 00:23:26.089485 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:26.089448 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef4c8094-7dr8j"] Nov 03 00:23:26.523732 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:26.523681 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 00:23:26.523926 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:26.523754 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" podUID="26ee8627-3e4b-486b-9e59-f646fa5f5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 00:23:26.711823 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:26.711783 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40d50c3-bdc7-443d-9c6b-0b438a20a6b7" path="/var/lib/kubelet/pods/e40d50c3-bdc7-443d-9c6b-0b438a20a6b7/volumes" Nov 03 00:23:27.523572 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:27.523533 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 00:23:27.523797 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:27.523601 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" podUID="26ee8627-3e4b-486b-9e59-f646fa5f5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 00:23:28.527476 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:28.527445 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:28.528011 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:28.527719 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:28.528341 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:28.528323 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" Nov 03 00:23:30.265825 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:30.265794 5672 scope.go:122] "RemoveContainer" containerID="4a571eb3d7bf81ac9bcb5f13051a51a7a0924c763e83c083bffbba9f6beb12b0" Nov 03 00:23:30.274391 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:30.274371 5672 scope.go:122] "RemoveContainer" containerID="5db183cb1818e3adba47d6f245a232476850a16cadafb1010eff655ddf6cb0c1" Nov 03 00:23:30.282335 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:30.282310 5672 scope.go:122] "RemoveContainer" containerID="281215eb20e0e81426abf4fc96b0ef7d551aee3d4663397a66cf5fc0e7a124cd" Nov 03 00:23:30.290162 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:30.290126 5672 scope.go:122] "RemoveContainer" containerID="0d2dd95cf25e00dc3f9a8a154970a54294ecec79d3800479cc2c168cd3685006" Nov 03 00:23:30.298176 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:30.298144 5672 scope.go:122] "RemoveContainer" containerID="7a19f8dd5d154aa1605c1c7275c4334d76215dd0358ec538e63f0b149875cb4f" Nov 03 00:23:30.306069 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:30.306045 5672 scope.go:122] "RemoveContainer" containerID="739dcacaadfac9e964fffb3704f79192f817f33bff79c7c38ed1103a5860d55c" Nov 03 00:23:32.540474 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:32.540438 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:23:33.875924 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.875873 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb"] Nov 03 00:23:33.881195 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.881166 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:33.947735 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.947679 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmqpq\" (UniqueName: \"kubernetes.io/projected/1e09ce1c-af58-4dd7-af07-389b72297c4b-kube-api-access-zmqpq\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:33.947735 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.947731 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:33.948021 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.947829 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:33.948021 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.947878 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:33.948021 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.947932 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:33.948021 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.947991 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:33.948021 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.948012 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:33.948383 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.948062 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:33.948383 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.948144 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e09ce1c-af58-4dd7-af07-389b72297c4b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:33.988550 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:33.988511 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb"] Nov 03 00:23:34.048791 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.048747 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e09ce1c-af58-4dd7-af07-389b72297c4b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.048791 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.048796 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmqpq\" (UniqueName: \"kubernetes.io/projected/1e09ce1c-af58-4dd7-af07-389b72297c4b-kube-api-access-zmqpq\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.049116 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.048831 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.049116 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.048945 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.049116 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.048995 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.049116 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.049046 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.049428 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.049171 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.049428 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.049203 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.049428 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.049262 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.049613 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.049497 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.049677 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.049633 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e09ce1c-af58-4dd7-af07-389b72297c4b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.050115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.049897 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.050115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.050003 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.050358 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.050207 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.052319 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.052287 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.052439 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.052359 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.103021 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.102989 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e09ce1c-af58-4dd7-af07-389b72297c4b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.103411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.103393 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmqpq\" (UniqueName: \"kubernetes.io/projected/1e09ce1c-af58-4dd7-af07-389b72297c4b-kube-api-access-zmqpq\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-5w5nb\" (UID: \"1e09ce1c-af58-4dd7-af07-389b72297c4b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.195279 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.195170 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:34.624036 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.624008 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb"] Nov 03 00:23:34.627398 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:23:34.627364 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e09ce1c_af58_4dd7_af07_389b72297c4b.slice/crio-f3a7dfa039d4d1cce21ad19d45e1b3dd30fda5da327245e4d8baf02797fe7101 WatchSource:0}: Error finding container f3a7dfa039d4d1cce21ad19d45e1b3dd30fda5da327245e4d8baf02797fe7101: Status 404 returned error can't find the container with id f3a7dfa039d4d1cce21ad19d45e1b3dd30fda5da327245e4d8baf02797fe7101 Nov 03 00:23:34.630606 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.630570 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:34.630703 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.630657 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:34.630747 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:34.630706 5672 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 00:23:35.453967 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:35.453929 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" event={"ID":"1e09ce1c-af58-4dd7-af07-389b72297c4b","Type":"ContainerStarted","Data":"d8832aa8d5940661aab925b37ad5a0e481018b3f75d5e6ca66282fac604edf0b"} Nov 03 00:23:35.454422 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:35.453988 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" event={"ID":"1e09ce1c-af58-4dd7-af07-389b72297c4b","Type":"ContainerStarted","Data":"f3a7dfa039d4d1cce21ad19d45e1b3dd30fda5da327245e4d8baf02797fe7101"} Nov 03 00:23:35.661217 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:35.661085 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podStartSLOduration=2.6610668090000003 podStartE2EDuration="2.661066809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:35.61383207 +0000 UTC m=+5409.021143922" watchObservedRunningTime="2026-11-03 00:23:35.661066809 +0000 UTC m=+5409.068378638" Nov 03 00:23:36.195857 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:36.195818 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:36.197364 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:36.197339 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:23:36.197428 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:36.197403 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podUID="1e09ce1c-af58-4dd7-af07-389b72297c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 00:23:37.196357 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:37.196317 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:23:37.196815 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:37.196386 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podUID="1e09ce1c-af58-4dd7-af07-389b72297c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 00:23:38.196510 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:38.196473 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:23:38.196913 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:38.196539 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podUID="1e09ce1c-af58-4dd7-af07-389b72297c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 00:23:39.196046 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:39.196012 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:23:39.196270 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:39.196067 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podUID="1e09ce1c-af58-4dd7-af07-389b72297c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 00:23:40.195697 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:40.195659 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:23:40.196124 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:40.195724 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podUID="1e09ce1c-af58-4dd7-af07-389b72297c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 00:23:41.196113 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:41.196076 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:23:41.196545 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:41.196131 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podUID="1e09ce1c-af58-4dd7-af07-389b72297c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 00:23:42.195631 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:42.195593 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 00:23:42.195822 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:42.195652 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podUID="1e09ce1c-af58-4dd7-af07-389b72297c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 00:23:44.197166 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:44.196427 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:23:44.197166 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:44.196528 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podUID="1e09ce1c-af58-4dd7-af07-389b72297c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:23:45.241369 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:45.241325 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:23:45.242036 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:45.242000 5672 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podUID="1e09ce1c-af58-4dd7-af07-389b72297c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:23:45.741975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:45.741941 5672 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:45.742291 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:45.742271 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:45.742912 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:23:45.742894 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" Nov 03 00:23:56.019716 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:23:56.019677 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:24:11.026503 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:24:11.026470 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:24:25.553097 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:24:25.553054 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:24:26.019986 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:24:26.019942 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:24:30.347025 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:24:30.346996 5672 scope.go:122] "RemoveContainer" containerID="d0d35f226d620b256b2d07c73ed989d6af99068c1966bfa9180c4138e2e9d040" Nov 03 00:24:30.355986 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:24:30.355955 5672 scope.go:122] "RemoveContainer" containerID="f2b526e2d3eafabec2e0029ac1377ff531adbb67657ae23f2134f56e6defc1d6" Nov 03 00:24:30.368071 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:24:30.368048 5672 scope.go:122] "RemoveContainer" containerID="9c8dcb7ee4d27ecb8d5765bdd79f021266b93ab1255e14efc3c37604288a4fa5" Nov 03 00:24:30.377089 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:24:30.377057 5672 scope.go:122] "RemoveContainer" containerID="6412016d944250b619db0ca7b7c9ba25210b57c6e7ddde028fb1174e92d90288" Nov 03 00:24:55.046503 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:24:55.046467 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg"] Nov 03 00:24:55.092644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:24:55.092609 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6cfdc2c-m9kxg"] Nov 03 00:24:56.018816 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:24:56.018775 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:24:56.712954 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:24:56.712917 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8807f9e-1791-40cb-887a-27c3f6c529a7" path="/var/lib/kubelet/pods/d8807f9e-1791-40cb-887a-27c3f6c529a7/volumes" Nov 03 00:24:57.143795 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:24:57.143762 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:25:07.043691 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:07.043613 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm"] Nov 03 00:25:07.089880 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:07.089850 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ba68c1-8fgjm"] Nov 03 00:25:08.711248 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:08.711209 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3111d9d4-1710-4ddc-aa22-a85a07b60894" path="/var/lib/kubelet/pods/3111d9d4-1710-4ddc-aa22-a85a07b60894/volumes" Nov 03 00:25:21.947064 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:21.947029 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:25:26.020037 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:25:26.019993 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:25:29.587180 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:29.586240 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:25:29.587180 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:29.586321 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" podUID="26ee8627-3e4b-486b-9e59-f646fa5f5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:32.317703 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:32.317668 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:25:44.575291 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:44.575249 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:25:44.575927 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:44.575894 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-gt4gv" podUID="26ee8627-3e4b-486b-9e59-f646fa5f5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:48.525340 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.525308 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr"] Nov 03 00:25:48.528731 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.528711 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:25:48.578156 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.578121 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c983cf86-7f8c-4279-a444-960d82deac39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr\" (UID: \"c983cf86-7f8c-4279-a444-960d82deac39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:25:48.578315 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.578198 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mdcr\" (UniqueName: \"kubernetes.io/projected/c983cf86-7f8c-4279-a444-960d82deac39-kube-api-access-6mdcr\") pod \"payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr\" (UID: \"c983cf86-7f8c-4279-a444-960d82deac39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:25:48.624332 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.624291 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-12945944-dockercfg-bc78r\"" Nov 03 00:25:48.624565 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.624540 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-adc00e\"" Nov 03 00:25:48.626837 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.626810 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr"] Nov 03 00:25:48.678821 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.678779 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c983cf86-7f8c-4279-a444-960d82deac39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr\" (UID: \"c983cf86-7f8c-4279-a444-960d82deac39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:25:48.679016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.678838 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mdcr\" (UniqueName: \"kubernetes.io/projected/c983cf86-7f8c-4279-a444-960d82deac39-kube-api-access-6mdcr\") pod \"payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr\" (UID: \"c983cf86-7f8c-4279-a444-960d82deac39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:25:48.679530 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.679508 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c983cf86-7f8c-4279-a444-960d82deac39-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr\" (UID: \"c983cf86-7f8c-4279-a444-960d82deac39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:25:48.687604 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.687564 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg"] Nov 03 00:25:48.691480 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.691457 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:25:48.733842 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.733804 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mdcr\" (UniqueName: \"kubernetes.io/projected/c983cf86-7f8c-4279-a444-960d82deac39-kube-api-access-6mdcr\") pod \"payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr\" (UID: \"c983cf86-7f8c-4279-a444-960d82deac39\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:25:48.746404 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.746352 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg"] Nov 03 00:25:48.779722 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.779631 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3e0307-9333-407f-b8c3-c540b73651fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg\" (UID: \"3c3e0307-9333-407f-b8c3-c540b73651fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:25:48.779722 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.779671 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnbbx\" (UniqueName: \"kubernetes.io/projected/3c3e0307-9333-407f-b8c3-c540b73651fe-kube-api-access-lnbbx\") pod \"payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg\" (UID: \"3c3e0307-9333-407f-b8c3-c540b73651fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:25:48.841522 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.841481 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:25:48.880861 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.880826 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3e0307-9333-407f-b8c3-c540b73651fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg\" (UID: \"3c3e0307-9333-407f-b8c3-c540b73651fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:25:48.880861 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.880861 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnbbx\" (UniqueName: \"kubernetes.io/projected/3c3e0307-9333-407f-b8c3-c540b73651fe-kube-api-access-lnbbx\") pod \"payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg\" (UID: \"3c3e0307-9333-407f-b8c3-c540b73651fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:25:48.881755 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.881730 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3e0307-9333-407f-b8c3-c540b73651fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg\" (UID: \"3c3e0307-9333-407f-b8c3-c540b73651fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:25:48.934472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:48.934441 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnbbx\" (UniqueName: \"kubernetes.io/projected/3c3e0307-9333-407f-b8c3-c540b73651fe-kube-api-access-lnbbx\") pod \"payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg\" (UID: \"3c3e0307-9333-407f-b8c3-c540b73651fe\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:25:49.005287 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:49.005260 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:25:49.209294 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:49.209270 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr"] Nov 03 00:25:49.211136 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:25:49.211108 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc983cf86_7f8c_4279_a444_960d82deac39.slice/crio-a69860deab78b0f71fbb655c13b55fa5aff713563983923ea52efa0dc6a2ac16 WatchSource:0}: Error finding container a69860deab78b0f71fbb655c13b55fa5aff713563983923ea52efa0dc6a2ac16: Status 404 returned error can't find the container with id a69860deab78b0f71fbb655c13b55fa5aff713563983923ea52efa0dc6a2ac16 Nov 03 00:25:49.374505 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:49.374481 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg"] Nov 03 00:25:49.376356 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:25:49.376324 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3e0307_9333_407f_b8c3_c540b73651fe.slice/crio-38a9500ab320878c22d0ff8f9b6ad9ffceb1cd6c0838f0bcf72b0eadb2f35e76 WatchSource:0}: Error finding container 38a9500ab320878c22d0ff8f9b6ad9ffceb1cd6c0838f0bcf72b0eadb2f35e76: Status 404 returned error can't find the container with id 38a9500ab320878c22d0ff8f9b6ad9ffceb1cd6c0838f0bcf72b0eadb2f35e76 Nov 03 00:25:50.210934 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:50.210891 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" event={"ID":"3c3e0307-9333-407f-b8c3-c540b73651fe","Type":"ContainerStarted","Data":"10e62d6b7f575c324fc6fa8339df6ab1101cea58b1b8c7c6b16eb8fa4df7fc11"} Nov 03 00:25:50.211411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:50.210954 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" event={"ID":"3c3e0307-9333-407f-b8c3-c540b73651fe","Type":"ContainerStarted","Data":"38a9500ab320878c22d0ff8f9b6ad9ffceb1cd6c0838f0bcf72b0eadb2f35e76"} Nov 03 00:25:50.211411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:50.211069 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:25:50.212493 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:50.212471 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" event={"ID":"c983cf86-7f8c-4279-a444-960d82deac39","Type":"ContainerStarted","Data":"8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae"} Nov 03 00:25:50.212607 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:50.212505 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" event={"ID":"c983cf86-7f8c-4279-a444-960d82deac39","Type":"ContainerStarted","Data":"a69860deab78b0f71fbb655c13b55fa5aff713563983923ea52efa0dc6a2ac16"} Nov 03 00:25:50.212607 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:50.212528 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:25:50.320087 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:50.320033 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" podStartSLOduration=2.320017191 podStartE2EDuration="2.320017191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:50.319389243 +0000 UTC m=+5543.726701097" watchObservedRunningTime="2026-11-03 00:25:50.320017191 +0000 UTC m=+5543.727329022" Nov 03 00:25:50.432190 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:25:50.432117 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" podStartSLOduration=2.432099093 podStartE2EDuration="2.432099093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:50.431328715 +0000 UTC m=+5543.838640542" watchObservedRunningTime="2026-11-03 00:25:50.432099093 +0000 UTC m=+5543.839410922" Nov 03 00:25:56.020382 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:25:56.020344 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:26:00.246853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:00.246810 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:26:01.220277 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:01.220250 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:26:01.220499 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:01.220398 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:26:18.160057 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:18.160021 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:26:26.020139 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:26:26.020101 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:26:30.417487 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:30.417457 5672 scope.go:122] "RemoveContainer" containerID="d7f864e793fb9d26030fba5b86cc1e3f8e25a8b9ad4a8415f52057aeef7de740" Nov 03 00:26:30.425848 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:30.425827 5672 scope.go:122] "RemoveContainer" containerID="d8109584b8e21538995aae98bf52a092ef8afd2899bb6441edcdc60dda613481" Nov 03 00:26:43.548844 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:43.548794 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:26:56.020135 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:26:56.020096 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:26:58.599642 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.599608 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96"] Nov 03 00:26:58.603224 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.603206 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:26:58.702718 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.702689 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:26:58.702912 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.702872 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:26:58.703034 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.702971 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:26:58.703102 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.702972 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:26:58.704595 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.704573 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96"] Nov 03 00:26:58.718694 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.718673 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000cea0c-xbb96\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:26:58.718814 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.718704 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000cea0c-xbb96\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:26:58.718814 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.718725 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slsdc\" (UniqueName: \"kubernetes.io/projected/b670ab33-9ad2-475a-b0f1-0512b642e748-kube-api-access-slsdc\") pod \"maas-api-revoke-sub-000cea0c-xbb96\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:26:58.820082 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.820056 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000cea0c-xbb96\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:26:58.820283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.820092 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000cea0c-xbb96\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:26:58.820283 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.820115 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slsdc\" (UniqueName: \"kubernetes.io/projected/b670ab33-9ad2-475a-b0f1-0512b642e748-kube-api-access-slsdc\") pod \"maas-api-revoke-sub-000cea0c-xbb96\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:26:58.820754 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.820730 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000cea0c-xbb96\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:26:58.822481 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.822458 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000cea0c-xbb96\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:26:58.874497 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.874443 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slsdc\" (UniqueName: \"kubernetes.io/projected/b670ab33-9ad2-475a-b0f1-0512b642e748-kube-api-access-slsdc\") pod \"maas-api-revoke-sub-000cea0c-xbb96\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:26:58.915577 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:58.915550 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:26:59.282360 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:59.282327 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96"] Nov 03 00:26:59.284014 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:26:59.283984 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb670ab33_9ad2_475a_b0f1_0512b642e748.slice/crio-eff9d03ff9c600f62d695aa8f9d659654b867326cbe83cfc8db8c69168ee0fa6 WatchSource:0}: Error finding container eff9d03ff9c600f62d695aa8f9d659654b867326cbe83cfc8db8c69168ee0fa6: Status 404 returned error can't find the container with id eff9d03ff9c600f62d695aa8f9d659654b867326cbe83cfc8db8c69168ee0fa6 Nov 03 00:26:59.579747 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:59.579668 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" event={"ID":"b670ab33-9ad2-475a-b0f1-0512b642e748","Type":"ContainerStarted","Data":"826db5cbdf3ec16725fae200c21d53eb16ed2f5275e7d6a6b854a55af23b1f2c"} Nov 03 00:26:59.579747 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:59.579719 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" event={"ID":"b670ab33-9ad2-475a-b0f1-0512b642e748","Type":"ContainerStarted","Data":"eff9d03ff9c600f62d695aa8f9d659654b867326cbe83cfc8db8c69168ee0fa6"} Nov 03 00:26:59.688826 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:26:59.688773 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" podStartSLOduration=1.688760818 podStartE2EDuration="1.688760818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:59.687266264 +0000 UTC m=+5613.094578094" watchObservedRunningTime="2026-11-03 00:26:59.688760818 +0000 UTC m=+5613.096072638" Nov 03 00:27:00.586877 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:00.586837 5672 generic.go:356] "Generic (PLEG): container finished" podID="b670ab33-9ad2-475a-b0f1-0512b642e748" containerID="826db5cbdf3ec16725fae200c21d53eb16ed2f5275e7d6a6b854a55af23b1f2c" exitCode=0 Nov 03 00:27:00.587098 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:00.586894 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" event={"ID":"b670ab33-9ad2-475a-b0f1-0512b642e748","Type":"ContainerDied","Data":"826db5cbdf3ec16725fae200c21d53eb16ed2f5275e7d6a6b854a55af23b1f2c"} Nov 03 00:27:01.726566 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:01.726537 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:27:01.849960 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:01.849882 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-openshift-ca\") pod \"b670ab33-9ad2-475a-b0f1-0512b642e748\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " Nov 03 00:27:01.850113 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:01.849978 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b670ab33-9ad2-475a-b0f1-0512b642e748-kube-api-access-slsdc\" (UniqueName: \"kubernetes.io/projected/b670ab33-9ad2-475a-b0f1-0512b642e748-kube-api-access-slsdc\") pod \"b670ab33-9ad2-475a-b0f1-0512b642e748\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " Nov 03 00:27:01.850113 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:01.850008 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-xks-ca\") pod \"b670ab33-9ad2-475a-b0f1-0512b642e748\" (UID: \"b670ab33-9ad2-475a-b0f1-0512b642e748\") " Nov 03 00:27:01.850293 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:01.850266 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-openshift-ca" pod "b670ab33-9ad2-475a-b0f1-0512b642e748" (UID: "b670ab33-9ad2-475a-b0f1-0512b642e748"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:27:01.852195 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:01.852164 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-xks-ca" pod "b670ab33-9ad2-475a-b0f1-0512b642e748" (UID: "b670ab33-9ad2-475a-b0f1-0512b642e748"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:27:01.852305 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:01.852249 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b670ab33-9ad2-475a-b0f1-0512b642e748-kube-api-access-slsdc" pod "b670ab33-9ad2-475a-b0f1-0512b642e748" (UID: "b670ab33-9ad2-475a-b0f1-0512b642e748"). InnerVolumeSpecName "kube-api-access-slsdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:27:01.950996 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:01.950957 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slsdc\" (UniqueName: \"kubernetes.io/projected/b670ab33-9ad2-475a-b0f1-0512b642e748-kube-api-access-slsdc\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:01.950996 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:01.950990 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:01.950996 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:01.951000 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b670ab33-9ad2-475a-b0f1-0512b642e748-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:02.597376 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:02.597334 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" event={"ID":"b670ab33-9ad2-475a-b0f1-0512b642e748","Type":"ContainerDied","Data":"eff9d03ff9c600f62d695aa8f9d659654b867326cbe83cfc8db8c69168ee0fa6"} Nov 03 00:27:02.597376 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:02.597373 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96" Nov 03 00:27:02.597589 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:02.597393 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eff9d03ff9c600f62d695aa8f9d659654b867326cbe83cfc8db8c69168ee0fa6" Nov 03 00:27:11.649225 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:11.649190 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:27:16.632095 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:16.632056 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg"] Nov 03 00:27:16.733345 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:16.733313 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr"] Nov 03 00:27:21.633121 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:21.633080 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" podUID="3c3e0307-9333-407f-b8c3-c540b73651fe" containerName="payload-processing" containerID="cri-o://10e62d6b7f575c324fc6fa8339df6ab1101cea58b1b8c7c6b16eb8fa4df7fc11" gracePeriod=25 Nov 03 00:27:21.733700 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:21.733644 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" podUID="c983cf86-7f8c-4279-a444-960d82deac39" containerName="payload-pre-processing" containerID="cri-o://8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae" gracePeriod=25 Nov 03 00:27:26.019819 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:27:26.019780 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:27:27.558126 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:27.558090 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:27:30.448125 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:30.448094 5672 scope.go:122] "RemoveContainer" containerID="9d4010a5aba2cffe8db064c0139c99c1bd3c776750cd10238342d9986f81c32d" Nov 03 00:27:31.218923 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:31.218888 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" podUID="c983cf86-7f8c-4279-a444-960d82deac39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:27:31.219141 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:31.218897 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" podUID="3c3e0307-9333-407f-b8c3-c540b73651fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:27:41.218247 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:41.218208 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" podUID="3c3e0307-9333-407f-b8c3-c540b73651fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:27:41.218675 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:41.218224 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" podUID="c983cf86-7f8c-4279-a444-960d82deac39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:27:41.791239 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:41.791200 5672 generic.go:356] "Generic (PLEG): container finished" podID="3c3e0307-9333-407f-b8c3-c540b73651fe" containerID="10e62d6b7f575c324fc6fa8339df6ab1101cea58b1b8c7c6b16eb8fa4df7fc11" exitCode=0 Nov 03 00:27:41.791405 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:41.791274 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" event={"ID":"3c3e0307-9333-407f-b8c3-c540b73651fe","Type":"ContainerDied","Data":"10e62d6b7f575c324fc6fa8339df6ab1101cea58b1b8c7c6b16eb8fa4df7fc11"} Nov 03 00:27:41.891042 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:41.890979 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:27:41.973870 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:41.973846 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:27:42.004804 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.004768 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3e0307-9333-407f-b8c3-c540b73651fe-kube-api-access-lnbbx\" (UniqueName: \"kubernetes.io/projected/3c3e0307-9333-407f-b8c3-c540b73651fe-kube-api-access-lnbbx\") pod \"3c3e0307-9333-407f-b8c3-c540b73651fe\" (UID: \"3c3e0307-9333-407f-b8c3-c540b73651fe\") " Nov 03 00:27:42.004971 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.004824 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3e0307-9333-407f-b8c3-c540b73651fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3e0307-9333-407f-b8c3-c540b73651fe-plugins-config-volume\") pod \"3c3e0307-9333-407f-b8c3-c540b73651fe\" (UID: \"3c3e0307-9333-407f-b8c3-c540b73651fe\") " Nov 03 00:27:42.005242 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.005219 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3e0307-9333-407f-b8c3-c540b73651fe-plugins-config-volume" pod "3c3e0307-9333-407f-b8c3-c540b73651fe" (UID: "3c3e0307-9333-407f-b8c3-c540b73651fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:27:42.006938 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.006911 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3e0307-9333-407f-b8c3-c540b73651fe-kube-api-access-lnbbx" pod "3c3e0307-9333-407f-b8c3-c540b73651fe" (UID: "3c3e0307-9333-407f-b8c3-c540b73651fe"). InnerVolumeSpecName "kube-api-access-lnbbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:27:42.105856 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.105828 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c983cf86-7f8c-4279-a444-960d82deac39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c983cf86-7f8c-4279-a444-960d82deac39-plugins-config-volume\") pod \"c983cf86-7f8c-4279-a444-960d82deac39\" (UID: \"c983cf86-7f8c-4279-a444-960d82deac39\") " Nov 03 00:27:42.106051 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.105911 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c983cf86-7f8c-4279-a444-960d82deac39-kube-api-access-6mdcr\" (UniqueName: \"kubernetes.io/projected/c983cf86-7f8c-4279-a444-960d82deac39-kube-api-access-6mdcr\") pod \"c983cf86-7f8c-4279-a444-960d82deac39\" (UID: \"c983cf86-7f8c-4279-a444-960d82deac39\") " Nov 03 00:27:42.106184 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.106166 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnbbx\" (UniqueName: \"kubernetes.io/projected/3c3e0307-9333-407f-b8c3-c540b73651fe-kube-api-access-lnbbx\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:42.106184 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.106185 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3e0307-9333-407f-b8c3-c540b73651fe-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:42.106389 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.106270 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c983cf86-7f8c-4279-a444-960d82deac39-plugins-config-volume" pod "c983cf86-7f8c-4279-a444-960d82deac39" (UID: "c983cf86-7f8c-4279-a444-960d82deac39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:27:42.108520 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.108494 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c983cf86-7f8c-4279-a444-960d82deac39-kube-api-access-6mdcr" pod "c983cf86-7f8c-4279-a444-960d82deac39" (UID: "c983cf86-7f8c-4279-a444-960d82deac39"). InnerVolumeSpecName "kube-api-access-6mdcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:27:42.207623 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.207556 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mdcr\" (UniqueName: \"kubernetes.io/projected/c983cf86-7f8c-4279-a444-960d82deac39-kube-api-access-6mdcr\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:42.207623 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.207580 5672 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c983cf86-7f8c-4279-a444-960d82deac39-plugins-config-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:27:42.797566 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.797535 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" Nov 03 00:27:42.797982 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.797529 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg" event={"ID":"3c3e0307-9333-407f-b8c3-c540b73651fe","Type":"ContainerDied","Data":"38a9500ab320878c22d0ff8f9b6ad9ffceb1cd6c0838f0bcf72b0eadb2f35e76"} Nov 03 00:27:42.797982 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.797677 5672 scope.go:122] "RemoveContainer" containerID="10e62d6b7f575c324fc6fa8339df6ab1101cea58b1b8c7c6b16eb8fa4df7fc11" Nov 03 00:27:42.799219 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.799108 5672 generic.go:356] "Generic (PLEG): container finished" podID="c983cf86-7f8c-4279-a444-960d82deac39" containerID="8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae" exitCode=0 Nov 03 00:27:42.799219 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.799168 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" Nov 03 00:27:42.799219 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.799182 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" event={"ID":"c983cf86-7f8c-4279-a444-960d82deac39","Type":"ContainerDied","Data":"8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae"} Nov 03 00:27:42.799434 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.799244 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr" event={"ID":"c983cf86-7f8c-4279-a444-960d82deac39","Type":"ContainerDied","Data":"a69860deab78b0f71fbb655c13b55fa5aff713563983923ea52efa0dc6a2ac16"} Nov 03 00:27:42.807173 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.807137 5672 scope.go:122] "RemoveContainer" containerID="8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae" Nov 03 00:27:42.814699 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.814667 5672 scope.go:122] "RemoveContainer" containerID="8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae" Nov 03 00:27:42.814952 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:27:42.814934 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae\": container with ID starting with 8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae not found: ID does not exist" containerID="8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae" Nov 03 00:27:42.815002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.814962 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae"} err="failed to get container status \"8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae\": rpc error: code = NotFound desc = could not find container \"8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae\": container with ID starting with 8c6ab524c87d2be4abcdec68e56ba2090cfea187461b54df42495280456aa9ae not found: ID does not exist" Nov 03 00:27:42.949476 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.949399 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg"] Nov 03 00:27:42.997062 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:42.997028 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-adc00e-6b8b44575nxgkg"] Nov 03 00:27:43.057372 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:43.057337 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr"] Nov 03 00:27:43.060319 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:43.060279 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-adc00e-5bb89jqxmr"] Nov 03 00:27:44.712340 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:44.712309 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3e0307-9333-407f-b8c3-c540b73651fe" path="/var/lib/kubelet/pods/3c3e0307-9333-407f-b8c3-c540b73651fe/volumes" Nov 03 00:27:44.712749 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:27:44.712676 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c983cf86-7f8c-4279-a444-960d82deac39" path="/var/lib/kubelet/pods/c983cf86-7f8c-4279-a444-960d82deac39/volumes" Nov 03 00:27:56.025595 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:27:56.025555 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:28:17.010100 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:28:17.010021 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:28:26.020311 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:28:26.020277 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:28:43.014517 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:28:43.014474 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:28:46.754762 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:28:46.754720 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:28:46.755365 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:28:46.755332 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podUID="1e09ce1c-af58-4dd7-af07-389b72297c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:28:56.019559 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:28:56.019518 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:29:09.011901 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:29:09.011864 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:29:23.126820 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:29:23.126784 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:29:26.066482 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:29:26.066441 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:29:33.716650 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:29:33.716602 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:29:45.807982 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:29:45.807944 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:29:56.019205 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:29:56.019169 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:30:04.211338 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:30:04.211303 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:30:26.020255 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:30:26.020217 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:30:56.019942 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:30:56.019904 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:30:57.629354 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:30:57.629320 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:31:26.020276 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:31:26.020199 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:31:56.020135 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:31:56.020092 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:32:02.044316 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:32:02.044278 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96"] Nov 03 00:32:02.090484 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:32:02.090454 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000cea0c-xbb96"] Nov 03 00:32:02.712073 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:32:02.712038 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b670ab33-9ad2-475a-b0f1-0512b642e748" path="/var/lib/kubelet/pods/b670ab33-9ad2-475a-b0f1-0512b642e748/volumes" Nov 03 00:32:26.019569 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:32:26.019532 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:32:52.843100 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:32:52.843065 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:32:56.019655 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:32:56.019612 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:33:03.814015 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:33:03.813980 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:33:22.543705 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:33:22.543659 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:33:26.020598 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:33:26.020551 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:33:30.517722 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:33:30.517687 5672 scope.go:122] "RemoveContainer" containerID="826db5cbdf3ec16725fae200c21d53eb16ed2f5275e7d6a6b854a55af23b1f2c" Nov 03 00:33:32.733119 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:33:32.733090 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:33:47.345810 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:33:47.345773 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:33:56.020010 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:33:56.019973 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:34:06.332235 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:34:06.332138 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:34:26.019105 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:34:26.019062 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:34:35.133718 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:34:35.133678 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:34:37.740173 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:34:37.739261 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:34:37.740173 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:34:37.739349 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:34:46.014745 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:34:46.014707 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:34:56.020244 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:34:56.020205 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:34:57.236811 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:34:57.236772 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:35:20.837415 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:35:20.837369 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:35:26.020369 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:35:26.020333 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:35:30.742305 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:35:30.742261 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:35:39.558872 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:35:39.558788 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:35:48.862588 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:35:48.862550 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:35:56.019621 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:35:56.019578 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:35:56.840512 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:35:56.840477 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:36:06.228504 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:36:06.228466 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:36:15.951639 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:36:15.951601 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:36:26.019396 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:36:26.019358 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:36:34.625357 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:36:34.625321 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:36:42.744108 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:36:42.744071 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:36:53.152978 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:36:53.152940 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:36:56.019510 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:36:56.019474 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:37:05.958369 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:37:05.958277 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:37:26.020482 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:37:26.020439 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:37:29.018244 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:37:29.018209 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:37:31.802334 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:37:31.802277 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:37:31.802899 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:37:31.802362 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-5w5nb" podUID="1e09ce1c-af58-4dd7-af07-389b72297c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:37:39.640620 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:37:39.640579 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:37:56.019591 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:37:56.019554 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:38:18.721465 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:38:18.720481 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:38:26.019934 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:38:26.019882 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:38:35.815483 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:38:35.815399 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:38:56.020284 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:38:56.020245 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:39:00.945444 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:00.945407 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:39:14.839373 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.839337 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m"] Nov 03 00:39:14.839774 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.839704 5672 cpu_manager.go:414] "containerMap: removing container" podUID="b670ab33-9ad2-475a-b0f1-0512b642e748" containerName="revoke-keys" Nov 03 00:39:14.839774 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.839715 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b670ab33-9ad2-475a-b0f1-0512b642e748" containerName="revoke-keys" Nov 03 00:39:14.839774 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.839730 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3c3e0307-9333-407f-b8c3-c540b73651fe" containerName="payload-processing" Nov 03 00:39:14.839774 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.839735 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3e0307-9333-407f-b8c3-c540b73651fe" containerName="payload-processing" Nov 03 00:39:14.839774 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.839752 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c983cf86-7f8c-4279-a444-960d82deac39" containerName="payload-pre-processing" Nov 03 00:39:14.839774 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.839758 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c983cf86-7f8c-4279-a444-960d82deac39" containerName="payload-pre-processing" Nov 03 00:39:14.839975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.839812 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3e0307-9333-407f-b8c3-c540b73651fe" containerName="payload-processing" Nov 03 00:39:14.839975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.839824 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c983cf86-7f8c-4279-a444-960d82deac39" containerName="payload-pre-processing" Nov 03 00:39:14.839975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.839834 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="b670ab33-9ad2-475a-b0f1-0512b642e748" containerName="revoke-keys" Nov 03 00:39:14.843063 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.843042 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:14.894324 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.894294 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:39:14.894324 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.894298 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:39:14.895628 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.895607 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m"] Nov 03 00:39:14.940251 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.940214 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:39:14.940420 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.940270 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:39:14.990968 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.990929 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8132403-6464-4299-8a17-b37752a523fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0b34aa-mhb6m\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:14.991178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.990990 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8132403-6464-4299-8a17-b37752a523fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0b34aa-mhb6m\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:14.991178 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:14.991090 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjhqg\" (UniqueName: \"kubernetes.io/projected/d8132403-6464-4299-8a17-b37752a523fe-kube-api-access-mjhqg\") pod \"maas-api-revoke-sub-cd0b34aa-mhb6m\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:15.091943 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:15.091846 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8132403-6464-4299-8a17-b37752a523fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0b34aa-mhb6m\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:15.091943 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:15.091912 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8132403-6464-4299-8a17-b37752a523fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0b34aa-mhb6m\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:15.092193 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:15.091959 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjhqg\" (UniqueName: \"kubernetes.io/projected/d8132403-6464-4299-8a17-b37752a523fe-kube-api-access-mjhqg\") pod \"maas-api-revoke-sub-cd0b34aa-mhb6m\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:15.092661 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:15.092641 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8132403-6464-4299-8a17-b37752a523fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0b34aa-mhb6m\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:15.094339 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:15.094308 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8132403-6464-4299-8a17-b37752a523fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0b34aa-mhb6m\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:15.145682 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:15.145649 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjhqg\" (UniqueName: \"kubernetes.io/projected/d8132403-6464-4299-8a17-b37752a523fe-kube-api-access-mjhqg\") pod \"maas-api-revoke-sub-cd0b34aa-mhb6m\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:15.155458 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:15.155430 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:15.518426 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:15.518399 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m"] Nov 03 00:39:15.520616 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:39:15.520587 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8132403_6464_4299_8a17_b37752a523fe.slice/crio-5ebdb93a5b520b73c44643503ea2288c094925fc3faaccf84868d90e28ccb6d6 WatchSource:0}: Error finding container 5ebdb93a5b520b73c44643503ea2288c094925fc3faaccf84868d90e28ccb6d6: Status 404 returned error can't find the container with id 5ebdb93a5b520b73c44643503ea2288c094925fc3faaccf84868d90e28ccb6d6 Nov 03 00:39:15.572122 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:15.572088 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" event={"ID":"d8132403-6464-4299-8a17-b37752a523fe","Type":"ContainerStarted","Data":"5ebdb93a5b520b73c44643503ea2288c094925fc3faaccf84868d90e28ccb6d6"} Nov 03 00:39:16.578336 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:16.578298 5672 generic.go:356] "Generic (PLEG): container finished" podID="d8132403-6464-4299-8a17-b37752a523fe" containerID="49bc9e7c923c40a20f2c3507ecad8054030ead134af713ac060d643de65c4d36" exitCode=0 Nov 03 00:39:16.578779 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:16.578381 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" event={"ID":"d8132403-6464-4299-8a17-b37752a523fe","Type":"ContainerDied","Data":"49bc9e7c923c40a20f2c3507ecad8054030ead134af713ac060d643de65c4d36"} Nov 03 00:39:17.719673 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:17.719646 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:17.821340 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:17.821306 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8132403-6464-4299-8a17-b37752a523fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8132403-6464-4299-8a17-b37752a523fe-maas-api-xks-ca\") pod \"d8132403-6464-4299-8a17-b37752a523fe\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " Nov 03 00:39:17.821517 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:17.821355 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8132403-6464-4299-8a17-b37752a523fe-kube-api-access-mjhqg\" (UniqueName: \"kubernetes.io/projected/d8132403-6464-4299-8a17-b37752a523fe-kube-api-access-mjhqg\") pod \"d8132403-6464-4299-8a17-b37752a523fe\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " Nov 03 00:39:17.821517 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:17.821398 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8132403-6464-4299-8a17-b37752a523fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8132403-6464-4299-8a17-b37752a523fe-maas-api-openshift-ca\") pod \"d8132403-6464-4299-8a17-b37752a523fe\" (UID: \"d8132403-6464-4299-8a17-b37752a523fe\") " Nov 03 00:39:17.821890 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:17.821856 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8132403-6464-4299-8a17-b37752a523fe-maas-api-openshift-ca" pod "d8132403-6464-4299-8a17-b37752a523fe" (UID: "d8132403-6464-4299-8a17-b37752a523fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:39:17.823447 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:17.823422 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8132403-6464-4299-8a17-b37752a523fe-maas-api-xks-ca" pod "d8132403-6464-4299-8a17-b37752a523fe" (UID: "d8132403-6464-4299-8a17-b37752a523fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:39:17.823561 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:17.823425 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8132403-6464-4299-8a17-b37752a523fe-kube-api-access-mjhqg" pod "d8132403-6464-4299-8a17-b37752a523fe" (UID: "d8132403-6464-4299-8a17-b37752a523fe"). InnerVolumeSpecName "kube-api-access-mjhqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:17.922445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:17.922355 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8132403-6464-4299-8a17-b37752a523fe-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:39:17.922445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:17.922389 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjhqg\" (UniqueName: \"kubernetes.io/projected/d8132403-6464-4299-8a17-b37752a523fe-kube-api-access-mjhqg\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:39:17.922445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:17.922402 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8132403-6464-4299-8a17-b37752a523fe-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:39:18.589870 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:18.589840 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" Nov 03 00:39:18.590064 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:18.589835 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m" event={"ID":"d8132403-6464-4299-8a17-b37752a523fe","Type":"ContainerDied","Data":"5ebdb93a5b520b73c44643503ea2288c094925fc3faaccf84868d90e28ccb6d6"} Nov 03 00:39:18.590064 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:18.589972 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ebdb93a5b520b73c44643503ea2288c094925fc3faaccf84868d90e28ccb6d6" Nov 03 00:39:26.020401 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:39:26.020363 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:39:29.849753 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:29.849716 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:39:45.644455 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:39:45.644374 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:39:56.019319 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:39:56.019280 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:40:26.019697 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:40:26.019613 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:40:33.057125 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.057081 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv"] Nov 03 00:40:33.057748 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.057325 5672 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" podUID="027227b3-6a47-4b4a-8506-c8bf8763f4d3" containerName="manager" containerID="cri-o://e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f" gracePeriod=10 Nov 03 00:40:33.311675 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.311621 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 03 00:40:33.403143 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.403105 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027227b3-6a47-4b4a-8506-c8bf8763f4d3-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/027227b3-6a47-4b4a-8506-c8bf8763f4d3-extensions-socket-volume\") pod \"027227b3-6a47-4b4a-8506-c8bf8763f4d3\" (UID: \"027227b3-6a47-4b4a-8506-c8bf8763f4d3\") " Nov 03 00:40:33.403325 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.403240 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027227b3-6a47-4b4a-8506-c8bf8763f4d3-kube-api-access-pcvkd\" (UniqueName: \"kubernetes.io/projected/027227b3-6a47-4b4a-8506-c8bf8763f4d3-kube-api-access-pcvkd\") pod \"027227b3-6a47-4b4a-8506-c8bf8763f4d3\" (UID: \"027227b3-6a47-4b4a-8506-c8bf8763f4d3\") " Nov 03 00:40:33.403529 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.403505 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027227b3-6a47-4b4a-8506-c8bf8763f4d3-extensions-socket-volume" pod "027227b3-6a47-4b4a-8506-c8bf8763f4d3" (UID: "027227b3-6a47-4b4a-8506-c8bf8763f4d3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:40:33.405360 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.405333 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027227b3-6a47-4b4a-8506-c8bf8763f4d3-kube-api-access-pcvkd" pod "027227b3-6a47-4b4a-8506-c8bf8763f4d3" (UID: "027227b3-6a47-4b4a-8506-c8bf8763f4d3"). InnerVolumeSpecName "kube-api-access-pcvkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:40:33.504779 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.504748 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcvkd\" (UniqueName: \"kubernetes.io/projected/027227b3-6a47-4b4a-8506-c8bf8763f4d3-kube-api-access-pcvkd\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:40:33.504779 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.504776 5672 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/027227b3-6a47-4b4a-8506-c8bf8763f4d3-extensions-socket-volume\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:40:33.975158 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.975109 5672 generic.go:356] "Generic (PLEG): container finished" podID="027227b3-6a47-4b4a-8506-c8bf8763f4d3" containerID="e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f" exitCode=0 Nov 03 00:40:33.975335 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.975169 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" Nov 03 00:40:33.975335 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.975194 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" event={"ID":"027227b3-6a47-4b4a-8506-c8bf8763f4d3","Type":"ContainerDied","Data":"e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f"} Nov 03 00:40:33.975335 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.975250 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv" event={"ID":"027227b3-6a47-4b4a-8506-c8bf8763f4d3","Type":"ContainerDied","Data":"69e676792526ea427b6b915b8e614d8646bd629b1ca202e7582c41112b78ecab"} Nov 03 00:40:33.975335 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.975275 5672 scope.go:122] "RemoveContainer" containerID="e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f" Nov 03 00:40:33.988347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.988325 5672 scope.go:122] "RemoveContainer" containerID="e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f" Nov 03 00:40:33.988603 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:40:33.988576 5672 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f\": container with ID starting with e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f not found: ID does not exist" containerID="e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f" Nov 03 00:40:33.988656 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:33.988606 5672 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f"} err="failed to get container status \"e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f\": rpc error: code = NotFound desc = could not find container \"e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f\": container with ID starting with e09ef4736aa6ec53af5098b4353c5482786e1e9281dec73f508c3a2f5bf29f5f not found: ID does not exist" Nov 03 00:40:34.128921 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:34.128891 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv"] Nov 03 00:40:34.175644 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:34.175617 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rqrwv"] Nov 03 00:40:34.710703 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:40:34.710668 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027227b3-6a47-4b4a-8506-c8bf8763f4d3" path="/var/lib/kubelet/pods/027227b3-6a47-4b4a-8506-c8bf8763f4d3/volumes" Nov 03 00:40:56.020448 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:40:56.020407 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:41:26.019626 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:41:26.019586 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:41:36.596483 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.596411 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj"] Nov 03 00:41:36.596934 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.596780 5672 cpu_manager.go:414] "containerMap: removing container" podUID="027227b3-6a47-4b4a-8506-c8bf8763f4d3" containerName="manager" Nov 03 00:41:36.596934 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.596792 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027227b3-6a47-4b4a-8506-c8bf8763f4d3" containerName="manager" Nov 03 00:41:36.596934 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.596805 5672 cpu_manager.go:414] "containerMap: removing container" podUID="d8132403-6464-4299-8a17-b37752a523fe" containerName="revoke-keys" Nov 03 00:41:36.596934 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.596811 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8132403-6464-4299-8a17-b37752a523fe" containerName="revoke-keys" Nov 03 00:41:36.596934 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.596883 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="027227b3-6a47-4b4a-8506-c8bf8763f4d3" containerName="manager" Nov 03 00:41:36.596934 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.596891 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8132403-6464-4299-8a17-b37752a523fe" containerName="revoke-keys" Nov 03 00:41:36.601004 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.600987 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" Nov 03 00:41:36.651350 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.651321 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj"] Nov 03 00:41:36.695929 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.695905 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-x7x5p\"" Nov 03 00:41:36.761333 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.761304 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5lp\" (UniqueName: \"kubernetes.io/projected/e8f30574-3996-4dc8-ba43-46c4c92d2bfe-kube-api-access-tc5lp\") pod \"kuadrant-operator-controller-manager-678d87766f-7t7hj\" (UID: \"e8f30574-3996-4dc8-ba43-46c4c92d2bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" Nov 03 00:41:36.761464 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.761367 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8f30574-3996-4dc8-ba43-46c4c92d2bfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7t7hj\" (UID: \"e8f30574-3996-4dc8-ba43-46c4c92d2bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" Nov 03 00:41:36.861890 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.861819 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5lp\" (UniqueName: \"kubernetes.io/projected/e8f30574-3996-4dc8-ba43-46c4c92d2bfe-kube-api-access-tc5lp\") pod \"kuadrant-operator-controller-manager-678d87766f-7t7hj\" (UID: \"e8f30574-3996-4dc8-ba43-46c4c92d2bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" Nov 03 00:41:36.862032 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.861902 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8f30574-3996-4dc8-ba43-46c4c92d2bfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7t7hj\" (UID: \"e8f30574-3996-4dc8-ba43-46c4c92d2bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" Nov 03 00:41:36.862294 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.862275 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8f30574-3996-4dc8-ba43-46c4c92d2bfe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7t7hj\" (UID: \"e8f30574-3996-4dc8-ba43-46c4c92d2bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" Nov 03 00:41:36.917656 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:36.917627 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5lp\" (UniqueName: \"kubernetes.io/projected/e8f30574-3996-4dc8-ba43-46c4c92d2bfe-kube-api-access-tc5lp\") pod \"kuadrant-operator-controller-manager-678d87766f-7t7hj\" (UID: \"e8f30574-3996-4dc8-ba43-46c4c92d2bfe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" Nov 03 00:41:37.213749 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:37.213654 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" Nov 03 00:41:37.685953 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:41:37.685926 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f30574_3996_4dc8_ba43_46c4c92d2bfe.slice/crio-9a5d4355b8304b6d9f14b78bdc139313c22e0bd3d739d9353dced1287873f4e5 WatchSource:0}: Error finding container 9a5d4355b8304b6d9f14b78bdc139313c22e0bd3d739d9353dced1287873f4e5: Status 404 returned error can't find the container with id 9a5d4355b8304b6d9f14b78bdc139313c22e0bd3d739d9353dced1287873f4e5 Nov 03 00:41:37.729915 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:37.729889 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj"] Nov 03 00:41:38.281066 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:38.281022 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" event={"ID":"e8f30574-3996-4dc8-ba43-46c4c92d2bfe","Type":"ContainerStarted","Data":"09dba3f709362e77d1f51fd32180829fbd6b3f8869cef048b1728f31c44d008c"} Nov 03 00:41:38.281275 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:38.281084 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" Nov 03 00:41:38.281275 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:38.281100 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" event={"ID":"e8f30574-3996-4dc8-ba43-46c4c92d2bfe","Type":"ContainerStarted","Data":"9a5d4355b8304b6d9f14b78bdc139313c22e0bd3d739d9353dced1287873f4e5"} Nov 03 00:41:38.483896 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:38.483837 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" podStartSLOduration=2.483819519 podStartE2EDuration="2.483819519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:38.436508481 +0000 UTC m=+6491.843820312" watchObservedRunningTime="2026-11-03 00:41:38.483819519 +0000 UTC m=+6491.891131350" Nov 03 00:41:49.286853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:49.286823 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7t7hj" Nov 03 00:41:56.021931 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:41:56.021892 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:41:58.633446 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:41:58.633408 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:42:15.918884 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:42:15.918850 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:42:22.739835 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:42:22.739262 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:42:22.739835 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:42:22.739352 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:42:24.959005 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:42:24.958972 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:42:26.020053 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:42:26.020016 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:42:32.946622 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:42:32.946579 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:42:50.037633 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:42:50.037595 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:42:52.747347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:42:52.747276 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:42:52.748016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:42:52.747964 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:42:56.020900 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:42:56.020860 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:43:01.871952 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:01.871916 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv"] Nov 03 00:43:01.875171 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:01.875135 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:01.931011 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:01.930979 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fec9a8b-nq5hv\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:01.931138 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:01.931085 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hndql\" (UniqueName: \"kubernetes.io/projected/5569383e-96f7-40ce-9656-65b6876637f8-kube-api-access-hndql\") pod \"maas-api-revoke-sub-5fec9a8b-nq5hv\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:01.931214 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:01.931170 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fec9a8b-nq5hv\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:01.973168 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:01.973112 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:43:01.973322 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:01.973198 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:43:01.973322 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:01.973213 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:43:01.973322 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:01.973207 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:43:01.974870 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:01.974850 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv"] Nov 03 00:43:02.031590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:02.031561 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hndql\" (UniqueName: \"kubernetes.io/projected/5569383e-96f7-40ce-9656-65b6876637f8-kube-api-access-hndql\") pod \"maas-api-revoke-sub-5fec9a8b-nq5hv\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:02.031724 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:02.031620 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fec9a8b-nq5hv\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:02.031724 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:02.031667 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fec9a8b-nq5hv\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:02.032298 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:02.032275 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fec9a8b-nq5hv\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:02.033960 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:02.033943 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fec9a8b-nq5hv\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:02.084512 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:02.084481 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hndql\" (UniqueName: \"kubernetes.io/projected/5569383e-96f7-40ce-9656-65b6876637f8-kube-api-access-hndql\") pod \"maas-api-revoke-sub-5fec9a8b-nq5hv\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:02.187700 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:02.187630 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:02.557296 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:02.557271 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv"] Nov 03 00:43:02.559834 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:43:02.559807 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5569383e_96f7_40ce_9656_65b6876637f8.slice/crio-54b258a1f1f8204abdaac6c7fb6f69d2843ce57ad23e1219f3a40381ab4bb9e4 WatchSource:0}: Error finding container 54b258a1f1f8204abdaac6c7fb6f69d2843ce57ad23e1219f3a40381ab4bb9e4: Status 404 returned error can't find the container with id 54b258a1f1f8204abdaac6c7fb6f69d2843ce57ad23e1219f3a40381ab4bb9e4 Nov 03 00:43:02.742197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:02.742143 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" event={"ID":"5569383e-96f7-40ce-9656-65b6876637f8","Type":"ContainerStarted","Data":"54b258a1f1f8204abdaac6c7fb6f69d2843ce57ad23e1219f3a40381ab4bb9e4"} Nov 03 00:43:03.749445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:03.749407 5672 generic.go:356] "Generic (PLEG): container finished" podID="5569383e-96f7-40ce-9656-65b6876637f8" containerID="448d7c03466fa7b7d1871ba4276fbf02c51ea99669914caa2bf26513767b4882" exitCode=0 Nov 03 00:43:03.749831 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:03.749489 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" event={"ID":"5569383e-96f7-40ce-9656-65b6876637f8","Type":"ContainerDied","Data":"448d7c03466fa7b7d1871ba4276fbf02c51ea99669914caa2bf26513767b4882"} Nov 03 00:43:04.886193 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:04.886167 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:04.956844 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:04.956809 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5569383e-96f7-40ce-9656-65b6876637f8-kube-api-access-hndql\" (UniqueName: \"kubernetes.io/projected/5569383e-96f7-40ce-9656-65b6876637f8-kube-api-access-hndql\") pod \"5569383e-96f7-40ce-9656-65b6876637f8\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " Nov 03 00:43:04.957003 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:04.956894 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-xks-ca\") pod \"5569383e-96f7-40ce-9656-65b6876637f8\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " Nov 03 00:43:04.957003 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:04.956922 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-openshift-ca\") pod \"5569383e-96f7-40ce-9656-65b6876637f8\" (UID: \"5569383e-96f7-40ce-9656-65b6876637f8\") " Nov 03 00:43:04.957358 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:04.957331 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-openshift-ca" pod "5569383e-96f7-40ce-9656-65b6876637f8" (UID: "5569383e-96f7-40ce-9656-65b6876637f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:04.958873 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:04.958850 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-xks-ca" pod "5569383e-96f7-40ce-9656-65b6876637f8" (UID: "5569383e-96f7-40ce-9656-65b6876637f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:04.958949 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:04.958902 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5569383e-96f7-40ce-9656-65b6876637f8-kube-api-access-hndql" pod "5569383e-96f7-40ce-9656-65b6876637f8" (UID: "5569383e-96f7-40ce-9656-65b6876637f8"). InnerVolumeSpecName "kube-api-access-hndql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:05.057918 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:05.057838 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hndql\" (UniqueName: \"kubernetes.io/projected/5569383e-96f7-40ce-9656-65b6876637f8-kube-api-access-hndql\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:43:05.057918 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:05.057865 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:43:05.057918 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:05.057875 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5569383e-96f7-40ce-9656-65b6876637f8-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:43:05.761563 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:05.761490 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" Nov 03 00:43:05.761720 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:05.761494 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv" event={"ID":"5569383e-96f7-40ce-9656-65b6876637f8","Type":"ContainerDied","Data":"54b258a1f1f8204abdaac6c7fb6f69d2843ce57ad23e1219f3a40381ab4bb9e4"} Nov 03 00:43:05.761720 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:05.761617 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54b258a1f1f8204abdaac6c7fb6f69d2843ce57ad23e1219f3a40381ab4bb9e4" Nov 03 00:43:12.634011 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.633965 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx"] Nov 03 00:43:12.634563 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.634354 5672 cpu_manager.go:414] "containerMap: removing container" podUID="5569383e-96f7-40ce-9656-65b6876637f8" containerName="revoke-keys" Nov 03 00:43:12.634563 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.634372 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5569383e-96f7-40ce-9656-65b6876637f8" containerName="revoke-keys" Nov 03 00:43:12.634563 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.634454 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="5569383e-96f7-40ce-9656-65b6876637f8" containerName="revoke-keys" Nov 03 00:43:12.638382 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.638350 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:12.688908 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.688473 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:43:12.732649 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.732612 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad5ca24-f7rdx\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:12.732857 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.732659 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvxp6\" (UniqueName: \"kubernetes.io/projected/a1459a01-67fa-4456-b3cc-933634508baa-kube-api-access-dvxp6\") pod \"maas-api-revoke-sub-9ad5ca24-f7rdx\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:12.732857 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.732824 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad5ca24-f7rdx\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:12.734746 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.734723 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:43:12.734890 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.734780 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:43:12.734936 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.734899 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:43:12.736660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.736639 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx"] Nov 03 00:43:12.833510 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.833474 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad5ca24-f7rdx\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:12.833701 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.833521 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad5ca24-f7rdx\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:12.833701 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.833552 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvxp6\" (UniqueName: \"kubernetes.io/projected/a1459a01-67fa-4456-b3cc-933634508baa-kube-api-access-dvxp6\") pod \"maas-api-revoke-sub-9ad5ca24-f7rdx\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:12.834246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.834219 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad5ca24-f7rdx\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:12.835779 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.835760 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad5ca24-f7rdx\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:12.887238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.887162 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvxp6\" (UniqueName: \"kubernetes.io/projected/a1459a01-67fa-4456-b3cc-933634508baa-kube-api-access-dvxp6\") pod \"maas-api-revoke-sub-9ad5ca24-f7rdx\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:12.950563 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:12.950530 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:13.315109 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:13.315079 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx"] Nov 03 00:43:13.317361 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:43:13.317326 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1459a01_67fa_4456_b3cc_933634508baa.slice/crio-f4c66e06d5fc630af228f6541386af92b427254be590ab186feb988d9e75c8d9 WatchSource:0}: Error finding container f4c66e06d5fc630af228f6541386af92b427254be590ab186feb988d9e75c8d9: Status 404 returned error can't find the container with id f4c66e06d5fc630af228f6541386af92b427254be590ab186feb988d9e75c8d9 Nov 03 00:43:13.803899 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:13.803862 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" event={"ID":"a1459a01-67fa-4456-b3cc-933634508baa","Type":"ContainerStarted","Data":"de826658fece8e2d635132e141b886338419ccf6b922e341e5ca312fe5919b44"} Nov 03 00:43:13.804339 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:13.803914 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" event={"ID":"a1459a01-67fa-4456-b3cc-933634508baa","Type":"ContainerStarted","Data":"f4c66e06d5fc630af228f6541386af92b427254be590ab186feb988d9e75c8d9"} Nov 03 00:43:13.911661 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:13.911551 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" podStartSLOduration=1.911538433 podStartE2EDuration="1.911538433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:13.909852338 +0000 UTC m=+6587.317164167" watchObservedRunningTime="2026-11-03 00:43:13.911538433 +0000 UTC m=+6587.318850263" Nov 03 00:43:14.810288 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:14.810256 5672 generic.go:356] "Generic (PLEG): container finished" podID="a1459a01-67fa-4456-b3cc-933634508baa" containerID="de826658fece8e2d635132e141b886338419ccf6b922e341e5ca312fe5919b44" exitCode=0 Nov 03 00:43:14.810690 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:14.810299 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" event={"ID":"a1459a01-67fa-4456-b3cc-933634508baa","Type":"ContainerDied","Data":"de826658fece8e2d635132e141b886338419ccf6b922e341e5ca312fe5919b44"} Nov 03 00:43:15.945362 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:15.945336 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:16.064885 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.064853 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1459a01-67fa-4456-b3cc-933634508baa-kube-api-access-dvxp6\" (UniqueName: \"kubernetes.io/projected/a1459a01-67fa-4456-b3cc-933634508baa-kube-api-access-dvxp6\") pod \"a1459a01-67fa-4456-b3cc-933634508baa\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " Nov 03 00:43:16.065073 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.064934 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-xks-ca\") pod \"a1459a01-67fa-4456-b3cc-933634508baa\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " Nov 03 00:43:16.065073 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.064993 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-openshift-ca\") pod \"a1459a01-67fa-4456-b3cc-933634508baa\" (UID: \"a1459a01-67fa-4456-b3cc-933634508baa\") " Nov 03 00:43:16.065370 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.065340 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-openshift-ca" pod "a1459a01-67fa-4456-b3cc-933634508baa" (UID: "a1459a01-67fa-4456-b3cc-933634508baa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:16.067111 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.067045 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1459a01-67fa-4456-b3cc-933634508baa-kube-api-access-dvxp6" pod "a1459a01-67fa-4456-b3cc-933634508baa" (UID: "a1459a01-67fa-4456-b3cc-933634508baa"). InnerVolumeSpecName "kube-api-access-dvxp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:16.067268 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.067131 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-xks-ca" pod "a1459a01-67fa-4456-b3cc-933634508baa" (UID: "a1459a01-67fa-4456-b3cc-933634508baa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:16.167061 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.166588 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:43:16.167061 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.166638 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1459a01-67fa-4456-b3cc-933634508baa-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:43:16.167061 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.166654 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvxp6\" (UniqueName: \"kubernetes.io/projected/a1459a01-67fa-4456-b3cc-933634508baa-kube-api-access-dvxp6\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:43:16.821520 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.821493 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" Nov 03 00:43:16.821695 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.821516 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx" event={"ID":"a1459a01-67fa-4456-b3cc-933634508baa","Type":"ContainerDied","Data":"f4c66e06d5fc630af228f6541386af92b427254be590ab186feb988d9e75c8d9"} Nov 03 00:43:16.821695 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:16.821566 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4c66e06d5fc630af228f6541386af92b427254be590ab186feb988d9e75c8d9" Nov 03 00:43:25.315584 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:25.315547 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:43:26.020015 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:43:26.019974 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:43:37.117884 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:37.117848 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:43:45.933693 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:45.933659 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8"] Nov 03 00:43:45.934088 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:45.934041 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a1459a01-67fa-4456-b3cc-933634508baa" containerName="revoke-keys" Nov 03 00:43:45.934088 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:45.934051 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1459a01-67fa-4456-b3cc-933634508baa" containerName="revoke-keys" Nov 03 00:43:45.934177 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:45.934118 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1459a01-67fa-4456-b3cc-933634508baa" containerName="revoke-keys" Nov 03 00:43:45.937535 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:45.937511 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:45.990210 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:45.990172 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:43:45.990366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:45.990224 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:43:45.992989 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:45.992951 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8"] Nov 03 00:43:46.036287 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.036265 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:43:46.036497 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.036478 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:43:46.043899 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.043882 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wspkt\" (UniqueName: \"kubernetes.io/projected/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-kube-api-access-wspkt\") pod \"maas-api-revoke-sub-32bb75e4-x4zt8\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:46.043975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.043914 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32bb75e4-x4zt8\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:46.043975 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.043945 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32bb75e4-x4zt8\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:46.145261 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.145236 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wspkt\" (UniqueName: \"kubernetes.io/projected/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-kube-api-access-wspkt\") pod \"maas-api-revoke-sub-32bb75e4-x4zt8\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:46.145405 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.145272 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32bb75e4-x4zt8\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:46.145405 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.145308 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32bb75e4-x4zt8\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:46.146007 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.145981 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32bb75e4-x4zt8\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:46.147518 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.147494 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32bb75e4-x4zt8\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:46.198221 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.198138 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wspkt\" (UniqueName: \"kubernetes.io/projected/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-kube-api-access-wspkt\") pod \"maas-api-revoke-sub-32bb75e4-x4zt8\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:46.255052 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.255023 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:46.618988 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.618963 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8"] Nov 03 00:43:46.621541 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:43:46.621513 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3c51b6_6520_4fc8_a630_2661eabcbeb7.slice/crio-7ecfdb535de35f0ee872144d8a16365afaf83da3b295abeb589d16097e3120aa WatchSource:0}: Error finding container 7ecfdb535de35f0ee872144d8a16365afaf83da3b295abeb589d16097e3120aa: Status 404 returned error can't find the container with id 7ecfdb535de35f0ee872144d8a16365afaf83da3b295abeb589d16097e3120aa Nov 03 00:43:46.989681 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.989649 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" event={"ID":"ee3c51b6-6520-4fc8-a630-2661eabcbeb7","Type":"ContainerStarted","Data":"ef1ad0bb24e2d10915095495d968b5243ba403ccb87c400baed30c0481efe2c2"} Nov 03 00:43:46.990180 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:46.989706 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" event={"ID":"ee3c51b6-6520-4fc8-a630-2661eabcbeb7","Type":"ContainerStarted","Data":"7ecfdb535de35f0ee872144d8a16365afaf83da3b295abeb589d16097e3120aa"} Nov 03 00:43:47.097913 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:47.097863 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" podStartSLOduration=2.097849406 podStartE2EDuration="2.097849406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:47.096085684 +0000 UTC m=+6620.503397513" watchObservedRunningTime="2026-11-03 00:43:47.097849406 +0000 UTC m=+6620.505161235" Nov 03 00:43:47.995857 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:47.995826 5672 generic.go:356] "Generic (PLEG): container finished" podID="ee3c51b6-6520-4fc8-a630-2661eabcbeb7" containerID="ef1ad0bb24e2d10915095495d968b5243ba403ccb87c400baed30c0481efe2c2" exitCode=0 Nov 03 00:43:47.996248 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:47.995902 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" event={"ID":"ee3c51b6-6520-4fc8-a630-2661eabcbeb7","Type":"ContainerDied","Data":"ef1ad0bb24e2d10915095495d968b5243ba403ccb87c400baed30c0481efe2c2"} Nov 03 00:43:49.129640 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:49.129613 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:49.173270 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:49.173242 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-openshift-ca\") pod \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " Nov 03 00:43:49.173391 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:49.173308 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-xks-ca\") pod \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " Nov 03 00:43:49.173391 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:49.173335 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-kube-api-access-wspkt\" (UniqueName: \"kubernetes.io/projected/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-kube-api-access-wspkt\") pod \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\" (UID: \"ee3c51b6-6520-4fc8-a630-2661eabcbeb7\") " Nov 03 00:43:49.173484 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:49.173461 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-openshift-ca" pod "ee3c51b6-6520-4fc8-a630-2661eabcbeb7" (UID: "ee3c51b6-6520-4fc8-a630-2661eabcbeb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:49.173591 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:49.173579 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:43:49.175376 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:49.175347 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-kube-api-access-wspkt" pod "ee3c51b6-6520-4fc8-a630-2661eabcbeb7" (UID: "ee3c51b6-6520-4fc8-a630-2661eabcbeb7"). InnerVolumeSpecName "kube-api-access-wspkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:49.175480 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:49.175386 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-xks-ca" pod "ee3c51b6-6520-4fc8-a630-2661eabcbeb7" (UID: "ee3c51b6-6520-4fc8-a630-2661eabcbeb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:49.273990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:49.273928 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:43:49.273990 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:49.273951 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wspkt\" (UniqueName: \"kubernetes.io/projected/ee3c51b6-6520-4fc8-a630-2661eabcbeb7-kube-api-access-wspkt\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:43:50.006858 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:50.006829 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" Nov 03 00:43:50.007055 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:50.006821 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8" event={"ID":"ee3c51b6-6520-4fc8-a630-2661eabcbeb7","Type":"ContainerDied","Data":"7ecfdb535de35f0ee872144d8a16365afaf83da3b295abeb589d16097e3120aa"} Nov 03 00:43:50.007055 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:50.006953 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ecfdb535de35f0ee872144d8a16365afaf83da3b295abeb589d16097e3120aa" Nov 03 00:43:56.019269 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:43:56.019226 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:43:58.563945 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:43:58.563909 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:44:10.263423 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:10.263383 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:44:18.045010 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:18.044978 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m"] Nov 03 00:44:18.091328 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:18.091299 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0b34aa-mhb6m"] Nov 03 00:44:18.710541 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:18.710508 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8132403-6464-4299-8a17-b37752a523fe" path="/var/lib/kubelet/pods/d8132403-6464-4299-8a17-b37752a523fe/volumes" Nov 03 00:44:26.020085 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:44:26.020042 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:44:28.531101 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.531068 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v"] Nov 03 00:44:28.531517 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.531474 5672 cpu_manager.go:414] "containerMap: removing container" podUID="ee3c51b6-6520-4fc8-a630-2661eabcbeb7" containerName="revoke-keys" Nov 03 00:44:28.531517 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.531487 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3c51b6-6520-4fc8-a630-2661eabcbeb7" containerName="revoke-keys" Nov 03 00:44:28.531597 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.531568 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3c51b6-6520-4fc8-a630-2661eabcbeb7" containerName="revoke-keys" Nov 03 00:44:28.534629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.534612 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:28.597284 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.597252 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v"] Nov 03 00:44:28.635187 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.635136 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:44:28.635357 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.635196 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:44:28.635357 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.635213 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:44:28.635357 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.635282 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:44:28.638747 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.638713 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-606b245a-25c4v\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:28.638830 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.638763 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-606b245a-25c4v\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:28.638876 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.638857 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rw8\" (UniqueName: \"kubernetes.io/projected/4ef68fda-9cb4-44b9-9660-da193343b1f4-kube-api-access-27rw8\") pod \"maas-api-revoke-sub-606b245a-25c4v\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:28.740095 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.740052 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-606b245a-25c4v\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:28.740307 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.740112 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rw8\" (UniqueName: \"kubernetes.io/projected/4ef68fda-9cb4-44b9-9660-da193343b1f4-kube-api-access-27rw8\") pod \"maas-api-revoke-sub-606b245a-25c4v\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:28.740307 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.740206 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-606b245a-25c4v\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:28.740823 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.740799 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-606b245a-25c4v\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:28.742397 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.742372 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-606b245a-25c4v\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:28.793669 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.793604 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rw8\" (UniqueName: \"kubernetes.io/projected/4ef68fda-9cb4-44b9-9660-da193343b1f4-kube-api-access-27rw8\") pod \"maas-api-revoke-sub-606b245a-25c4v\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:28.847262 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:28.847219 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:29.214044 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:29.214016 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v"] Nov 03 00:44:29.216542 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:44:29.216476 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef68fda_9cb4_44b9_9660_da193343b1f4.slice/crio-229942646b318cc3420ae8bd2d6f7fffb9f00c10325f96fd91139a33e6d55560 WatchSource:0}: Error finding container 229942646b318cc3420ae8bd2d6f7fffb9f00c10325f96fd91139a33e6d55560: Status 404 returned error can't find the container with id 229942646b318cc3420ae8bd2d6f7fffb9f00c10325f96fd91139a33e6d55560 Nov 03 00:44:29.239657 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:29.239631 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" event={"ID":"4ef68fda-9cb4-44b9-9660-da193343b1f4","Type":"ContainerStarted","Data":"229942646b318cc3420ae8bd2d6f7fffb9f00c10325f96fd91139a33e6d55560"} Nov 03 00:44:30.246557 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:30.246522 5672 generic.go:356] "Generic (PLEG): container finished" podID="4ef68fda-9cb4-44b9-9660-da193343b1f4" containerID="bd779474759edcff268c90ef7f2b52db8b8829b6dc74d073b80a0f52f93975da" exitCode=0 Nov 03 00:44:30.246993 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:30.246615 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" event={"ID":"4ef68fda-9cb4-44b9-9660-da193343b1f4","Type":"ContainerDied","Data":"bd779474759edcff268c90ef7f2b52db8b8829b6dc74d073b80a0f52f93975da"} Nov 03 00:44:31.392468 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:31.392441 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:31.463475 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:31.463449 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-openshift-ca\") pod \"4ef68fda-9cb4-44b9-9660-da193343b1f4\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " Nov 03 00:44:31.463599 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:31.463522 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef68fda-9cb4-44b9-9660-da193343b1f4-kube-api-access-27rw8\" (UniqueName: \"kubernetes.io/projected/4ef68fda-9cb4-44b9-9660-da193343b1f4-kube-api-access-27rw8\") pod \"4ef68fda-9cb4-44b9-9660-da193343b1f4\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " Nov 03 00:44:31.463599 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:31.463574 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-xks-ca\") pod \"4ef68fda-9cb4-44b9-9660-da193343b1f4\" (UID: \"4ef68fda-9cb4-44b9-9660-da193343b1f4\") " Nov 03 00:44:31.463834 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:31.463812 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-openshift-ca" pod "4ef68fda-9cb4-44b9-9660-da193343b1f4" (UID: "4ef68fda-9cb4-44b9-9660-da193343b1f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:31.465628 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:31.465599 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-xks-ca" pod "4ef68fda-9cb4-44b9-9660-da193343b1f4" (UID: "4ef68fda-9cb4-44b9-9660-da193343b1f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:31.465628 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:31.465611 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef68fda-9cb4-44b9-9660-da193343b1f4-kube-api-access-27rw8" pod "4ef68fda-9cb4-44b9-9660-da193343b1f4" (UID: "4ef68fda-9cb4-44b9-9660-da193343b1f4"). InnerVolumeSpecName "kube-api-access-27rw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:31.564576 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:31.564505 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:44:31.564576 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:31.564529 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rw8\" (UniqueName: \"kubernetes.io/projected/4ef68fda-9cb4-44b9-9660-da193343b1f4-kube-api-access-27rw8\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:44:31.564576 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:31.564539 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ef68fda-9cb4-44b9-9660-da193343b1f4-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:44:32.258538 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:32.258508 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" Nov 03 00:44:32.258538 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:32.258534 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v" event={"ID":"4ef68fda-9cb4-44b9-9660-da193343b1f4","Type":"ContainerDied","Data":"229942646b318cc3420ae8bd2d6f7fffb9f00c10325f96fd91139a33e6d55560"} Nov 03 00:44:32.258755 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:32.258575 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="229942646b318cc3420ae8bd2d6f7fffb9f00c10325f96fd91139a33e6d55560" Nov 03 00:44:39.341337 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.341253 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw"] Nov 03 00:44:39.341729 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.341613 5672 cpu_manager.go:414] "containerMap: removing container" podUID="4ef68fda-9cb4-44b9-9660-da193343b1f4" containerName="revoke-keys" Nov 03 00:44:39.341729 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.341625 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef68fda-9cb4-44b9-9660-da193343b1f4" containerName="revoke-keys" Nov 03 00:44:39.341729 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.341711 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef68fda-9cb4-44b9-9660-da193343b1f4" containerName="revoke-keys" Nov 03 00:44:39.344942 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.344925 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:39.432864 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.432830 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a9bfad0-4z7jw\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:39.433037 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.432876 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a9bfad0-4z7jw\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:39.433037 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.432951 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqcwf\" (UniqueName: \"kubernetes.io/projected/738b33f6-0402-4204-8d80-bd7cdc0ab763-kube-api-access-gqcwf\") pod \"maas-api-revoke-sub-8a9bfad0-4z7jw\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:39.442327 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.442299 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:44:39.442487 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.442348 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:44:39.442487 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.442385 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:44:39.442487 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.442430 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:44:39.444037 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.444016 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw"] Nov 03 00:44:39.534271 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.534237 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a9bfad0-4z7jw\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:39.534470 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.534280 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a9bfad0-4z7jw\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:39.534470 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.534305 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqcwf\" (UniqueName: \"kubernetes.io/projected/738b33f6-0402-4204-8d80-bd7cdc0ab763-kube-api-access-gqcwf\") pod \"maas-api-revoke-sub-8a9bfad0-4z7jw\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:39.534976 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.534947 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a9bfad0-4z7jw\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:39.536617 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.536595 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a9bfad0-4z7jw\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:39.588008 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.587978 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqcwf\" (UniqueName: \"kubernetes.io/projected/738b33f6-0402-4204-8d80-bd7cdc0ab763-kube-api-access-gqcwf\") pod \"maas-api-revoke-sub-8a9bfad0-4z7jw\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:39.660272 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:39.660178 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:40.027700 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:40.027670 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw"] Nov 03 00:44:40.029828 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:44:40.029798 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738b33f6_0402_4204_8d80_bd7cdc0ab763.slice/crio-a9673eecb8fbc56f2e4878e04bc047df09d487226e5a84f2fcfcfb8386686022 WatchSource:0}: Error finding container a9673eecb8fbc56f2e4878e04bc047df09d487226e5a84f2fcfcfb8386686022: Status 404 returned error can't find the container with id a9673eecb8fbc56f2e4878e04bc047df09d487226e5a84f2fcfcfb8386686022 Nov 03 00:44:40.305730 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:40.305642 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" event={"ID":"738b33f6-0402-4204-8d80-bd7cdc0ab763","Type":"ContainerStarted","Data":"a9673eecb8fbc56f2e4878e04bc047df09d487226e5a84f2fcfcfb8386686022"} Nov 03 00:44:41.312771 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:41.312679 5672 generic.go:356] "Generic (PLEG): container finished" podID="738b33f6-0402-4204-8d80-bd7cdc0ab763" containerID="7d3c1f1aef9c483f281bddaf48c81361afdac58467d43c4f577cfd034ae16a28" exitCode=0 Nov 03 00:44:41.312771 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:41.312735 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" event={"ID":"738b33f6-0402-4204-8d80-bd7cdc0ab763","Type":"ContainerDied","Data":"7d3c1f1aef9c483f281bddaf48c81361afdac58467d43c4f577cfd034ae16a28"} Nov 03 00:44:41.349914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:41.349882 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:44:42.449706 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:42.449679 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:42.561306 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:42.561274 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-openshift-ca\") pod \"738b33f6-0402-4204-8d80-bd7cdc0ab763\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " Nov 03 00:44:42.561489 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:42.561356 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-xks-ca\") pod \"738b33f6-0402-4204-8d80-bd7cdc0ab763\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " Nov 03 00:44:42.561489 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:42.561424 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738b33f6-0402-4204-8d80-bd7cdc0ab763-kube-api-access-gqcwf\" (UniqueName: \"kubernetes.io/projected/738b33f6-0402-4204-8d80-bd7cdc0ab763-kube-api-access-gqcwf\") pod \"738b33f6-0402-4204-8d80-bd7cdc0ab763\" (UID: \"738b33f6-0402-4204-8d80-bd7cdc0ab763\") " Nov 03 00:44:42.561998 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:42.561967 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-openshift-ca" pod "738b33f6-0402-4204-8d80-bd7cdc0ab763" (UID: "738b33f6-0402-4204-8d80-bd7cdc0ab763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:42.564341 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:42.564309 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738b33f6-0402-4204-8d80-bd7cdc0ab763-kube-api-access-gqcwf" pod "738b33f6-0402-4204-8d80-bd7cdc0ab763" (UID: "738b33f6-0402-4204-8d80-bd7cdc0ab763"). InnerVolumeSpecName "kube-api-access-gqcwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:42.566966 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:42.566940 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-xks-ca" pod "738b33f6-0402-4204-8d80-bd7cdc0ab763" (UID: "738b33f6-0402-4204-8d80-bd7cdc0ab763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:42.662518 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:42.662484 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:44:42.662518 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:42.662513 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqcwf\" (UniqueName: \"kubernetes.io/projected/738b33f6-0402-4204-8d80-bd7cdc0ab763-kube-api-access-gqcwf\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:44:42.662518 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:42.662522 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/738b33f6-0402-4204-8d80-bd7cdc0ab763-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:44:43.323911 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:43.323875 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" Nov 03 00:44:43.324094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:43.323906 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw" event={"ID":"738b33f6-0402-4204-8d80-bd7cdc0ab763","Type":"ContainerDied","Data":"a9673eecb8fbc56f2e4878e04bc047df09d487226e5a84f2fcfcfb8386686022"} Nov 03 00:44:43.324094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:43.323958 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9673eecb8fbc56f2e4878e04bc047df09d487226e5a84f2fcfcfb8386686022" Nov 03 00:44:52.039056 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:44:52.039020 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:44:56.019809 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:44:56.019767 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:45:00.154469 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.154430 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx"] Nov 03 00:45:00.155779 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.155757 5672 cpu_manager.go:414] "containerMap: removing container" podUID="738b33f6-0402-4204-8d80-bd7cdc0ab763" containerName="revoke-keys" Nov 03 00:45:00.155949 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.155931 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="738b33f6-0402-4204-8d80-bd7cdc0ab763" containerName="revoke-keys" Nov 03 00:45:00.156253 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.156222 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="738b33f6-0402-4204-8d80-bd7cdc0ab763" containerName="revoke-keys" Nov 03 00:45:00.165871 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.165853 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" Nov 03 00:45:00.181826 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.181786 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d24ff4f-182b-4b9e-b374-a77621a01f5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-qmwsx\" (UID: \"3d24ff4f-182b-4b9e-b374-a77621a01f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" Nov 03 00:45:00.182092 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.182069 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767cw\" (UniqueName: \"kubernetes.io/projected/3d24ff4f-182b-4b9e-b374-a77621a01f5e-kube-api-access-767cw\") pod \"maas-api-key-cleanup-29894445-qmwsx\" (UID: \"3d24ff4f-182b-4b9e-b374-a77621a01f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" Nov 03 00:45:00.212503 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.212470 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx"] Nov 03 00:45:00.263590 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.263557 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:45:00.264822 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.264777 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:45:00.265046 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.265030 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:45:00.291195 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.284608 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d24ff4f-182b-4b9e-b374-a77621a01f5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-qmwsx\" (UID: \"3d24ff4f-182b-4b9e-b374-a77621a01f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" Nov 03 00:45:00.291195 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.284658 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767cw\" (UniqueName: \"kubernetes.io/projected/3d24ff4f-182b-4b9e-b374-a77621a01f5e-kube-api-access-767cw\") pod \"maas-api-key-cleanup-29894445-qmwsx\" (UID: \"3d24ff4f-182b-4b9e-b374-a77621a01f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" Nov 03 00:45:00.297191 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.292186 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d24ff4f-182b-4b9e-b374-a77621a01f5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-qmwsx\" (UID: \"3d24ff4f-182b-4b9e-b374-a77621a01f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" Nov 03 00:45:00.390172 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.389049 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-767cw\" (UniqueName: \"kubernetes.io/projected/3d24ff4f-182b-4b9e-b374-a77621a01f5e-kube-api-access-767cw\") pod \"maas-api-key-cleanup-29894445-qmwsx\" (UID: \"3d24ff4f-182b-4b9e-b374-a77621a01f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" Nov 03 00:45:00.506521 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.506325 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" Nov 03 00:45:00.984539 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:00.973296 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx"] Nov 03 00:45:01.434119 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:01.434082 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" event={"ID":"3d24ff4f-182b-4b9e-b374-a77621a01f5e","Type":"ContainerStarted","Data":"1bb50898372f6780d10a35e80f2602bba0c6de5cb9e5569e9f87e93b6db46f75"} Nov 03 00:45:02.440586 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:02.440549 5672 generic.go:356] "Generic (PLEG): container finished" podID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerID="2c24f0593d86130e8960e1ee765fcc09c976fd838d1da176be24d221636c80b6" exitCode=60 Nov 03 00:45:02.441017 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:02.440634 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" event={"ID":"3d24ff4f-182b-4b9e-b374-a77621a01f5e","Type":"ContainerDied","Data":"2c24f0593d86130e8960e1ee765fcc09c976fd838d1da176be24d221636c80b6"} Nov 03 00:45:02.441017 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:02.440807 5672 scope.go:122] "RemoveContainer" containerID="2c24f0593d86130e8960e1ee765fcc09c976fd838d1da176be24d221636c80b6" Nov 03 00:45:03.447378 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:03.447342 5672 generic.go:356] "Generic (PLEG): container finished" podID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerID="9843480c4fc71f58dec1fc865f26fa853be1ae7329f9716c11bde5eeaa60ba71" exitCode=60 Nov 03 00:45:03.447797 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:03.447426 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" event={"ID":"3d24ff4f-182b-4b9e-b374-a77621a01f5e","Type":"ContainerDied","Data":"9843480c4fc71f58dec1fc865f26fa853be1ae7329f9716c11bde5eeaa60ba71"} Nov 03 00:45:03.447797 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:03.447487 5672 scope.go:122] "RemoveContainer" containerID="2c24f0593d86130e8960e1ee765fcc09c976fd838d1da176be24d221636c80b6" Nov 03 00:45:03.447797 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:03.447751 5672 scope.go:122] "RemoveContainer" containerID="9843480c4fc71f58dec1fc865f26fa853be1ae7329f9716c11bde5eeaa60ba71" Nov 03 00:45:03.447990 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:45:03.447971 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-qmwsx_odh-ai-gateway-infra(3d24ff4f-182b-4b9e-b374-a77621a01f5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" Nov 03 00:45:04.354642 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:04.354609 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:45:04.454122 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:04.454097 5672 scope.go:122] "RemoveContainer" containerID="9843480c4fc71f58dec1fc865f26fa853be1ae7329f9716c11bde5eeaa60ba71" Nov 03 00:45:04.454518 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:45:04.454288 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-qmwsx_odh-ai-gateway-infra(3d24ff4f-182b-4b9e-b374-a77621a01f5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" Nov 03 00:45:16.637387 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:16.637346 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:45:17.705893 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:17.705848 5672 scope.go:122] "RemoveContainer" containerID="9843480c4fc71f58dec1fc865f26fa853be1ae7329f9716c11bde5eeaa60ba71" Nov 03 00:45:18.562526 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:18.562482 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" event={"ID":"3d24ff4f-182b-4b9e-b374-a77621a01f5e","Type":"ContainerStarted","Data":"12128bda7697fd25e8c84333b2cd7e64f5421332ca76997caeafa06a5c2f5337"} Nov 03 00:45:18.673133 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:18.671000 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" podStartSLOduration=18.670980423 podStartE2EDuration="18.670980423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:18.669533534 +0000 UTC m=+6712.076845367" watchObservedRunningTime="2026-11-03 00:45:18.670980423 +0000 UTC m=+6712.078292253" Nov 03 00:45:19.592650 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:19.592615 5672 generic.go:356] "Generic (PLEG): container finished" podID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerID="12128bda7697fd25e8c84333b2cd7e64f5421332ca76997caeafa06a5c2f5337" exitCode=60 Nov 03 00:45:19.593384 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:19.593358 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" event={"ID":"3d24ff4f-182b-4b9e-b374-a77621a01f5e","Type":"ContainerDied","Data":"12128bda7697fd25e8c84333b2cd7e64f5421332ca76997caeafa06a5c2f5337"} Nov 03 00:45:19.593570 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:19.593553 5672 scope.go:122] "RemoveContainer" containerID="9843480c4fc71f58dec1fc865f26fa853be1ae7329f9716c11bde5eeaa60ba71" Nov 03 00:45:19.594124 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:19.594106 5672 scope.go:122] "RemoveContainer" containerID="12128bda7697fd25e8c84333b2cd7e64f5421332ca76997caeafa06a5c2f5337" Nov 03 00:45:19.594565 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:45:19.594537 5672 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-qmwsx_odh-ai-gateway-infra(3d24ff4f-182b-4b9e-b374-a77621a01f5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" Nov 03 00:45:19.707478 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:19.707442 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx"] Nov 03 00:45:20.729937 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:20.729910 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" Nov 03 00:45:20.896014 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:20.895904 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d24ff4f-182b-4b9e-b374-a77621a01f5e-kube-api-access-767cw\" (UniqueName: \"kubernetes.io/projected/3d24ff4f-182b-4b9e-b374-a77621a01f5e-kube-api-access-767cw\") pod \"3d24ff4f-182b-4b9e-b374-a77621a01f5e\" (UID: \"3d24ff4f-182b-4b9e-b374-a77621a01f5e\") " Nov 03 00:45:20.896014 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:20.895970 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d24ff4f-182b-4b9e-b374-a77621a01f5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d24ff4f-182b-4b9e-b374-a77621a01f5e-maas-api-service-ca\") pod \"3d24ff4f-182b-4b9e-b374-a77621a01f5e\" (UID: \"3d24ff4f-182b-4b9e-b374-a77621a01f5e\") " Nov 03 00:45:20.896373 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:20.896352 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d24ff4f-182b-4b9e-b374-a77621a01f5e-maas-api-service-ca" pod "3d24ff4f-182b-4b9e-b374-a77621a01f5e" (UID: "3d24ff4f-182b-4b9e-b374-a77621a01f5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:20.897968 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:20.897939 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d24ff4f-182b-4b9e-b374-a77621a01f5e-kube-api-access-767cw" pod "3d24ff4f-182b-4b9e-b374-a77621a01f5e" (UID: "3d24ff4f-182b-4b9e-b374-a77621a01f5e"). InnerVolumeSpecName "kube-api-access-767cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:20.997439 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:20.997401 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-767cw\" (UniqueName: \"kubernetes.io/projected/3d24ff4f-182b-4b9e-b374-a77621a01f5e-kube-api-access-767cw\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:45:20.997439 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:20.997432 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d24ff4f-182b-4b9e-b374-a77621a01f5e-maas-api-service-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:45:21.606388 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:21.606354 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" Nov 03 00:45:21.606599 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:21.606352 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx" event={"ID":"3d24ff4f-182b-4b9e-b374-a77621a01f5e","Type":"ContainerDied","Data":"1bb50898372f6780d10a35e80f2602bba0c6de5cb9e5569e9f87e93b6db46f75"} Nov 03 00:45:21.606599 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:21.606505 5672 scope.go:122] "RemoveContainer" containerID="12128bda7697fd25e8c84333b2cd7e64f5421332ca76997caeafa06a5c2f5337" Nov 03 00:45:21.759675 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:21.759628 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx"] Nov 03 00:45:21.763254 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:21.763222 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-qmwsx"] Nov 03 00:45:22.711695 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:22.711647 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" path="/var/lib/kubelet/pods/3d24ff4f-182b-4b9e-b374-a77621a01f5e/volumes" Nov 03 00:45:26.020170 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:45:26.020109 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:45:27.231205 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.231167 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl"] Nov 03 00:45:27.231627 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.231539 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerName="cleanup" Nov 03 00:45:27.231627 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.231550 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerName="cleanup" Nov 03 00:45:27.231627 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.231569 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerName="cleanup" Nov 03 00:45:27.231627 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.231574 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerName="cleanup" Nov 03 00:45:27.231766 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.231637 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerName="cleanup" Nov 03 00:45:27.231766 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.231646 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerName="cleanup" Nov 03 00:45:27.236140 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.236119 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:27.288165 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.288115 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl"] Nov 03 00:45:27.332096 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.332056 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:45:27.332096 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.332095 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:45:27.332352 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.332121 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:45:27.332352 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.332166 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:45:27.351661 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.351631 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-635820af-dx2tl\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:27.351769 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.351695 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-635820af-dx2tl\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:27.351824 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.351801 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szd4x\" (UniqueName: \"kubernetes.io/projected/5e0059da-2321-4ed9-91f3-11af0d82efa7-kube-api-access-szd4x\") pod \"maas-api-revoke-sub-635820af-dx2tl\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:27.452985 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.452951 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szd4x\" (UniqueName: \"kubernetes.io/projected/5e0059da-2321-4ed9-91f3-11af0d82efa7-kube-api-access-szd4x\") pod \"maas-api-revoke-sub-635820af-dx2tl\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:27.453213 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.453020 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-635820af-dx2tl\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:27.453213 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.453067 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-635820af-dx2tl\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:27.453776 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.453752 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-635820af-dx2tl\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:27.455368 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.455342 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-635820af-dx2tl\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:27.506851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.506780 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szd4x\" (UniqueName: \"kubernetes.io/projected/5e0059da-2321-4ed9-91f3-11af0d82efa7-kube-api-access-szd4x\") pod \"maas-api-revoke-sub-635820af-dx2tl\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:27.548342 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.548306 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:27.911334 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:27.911201 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl"] Nov 03 00:45:27.914385 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:45:27.914336 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0059da_2321_4ed9_91f3_11af0d82efa7.slice/crio-98d8c2fe29dbcdaadf8ec578d876b806adfe026a89a927f9a091a42cfef8542b WatchSource:0}: Error finding container 98d8c2fe29dbcdaadf8ec578d876b806adfe026a89a927f9a091a42cfef8542b: Status 404 returned error can't find the container with id 98d8c2fe29dbcdaadf8ec578d876b806adfe026a89a927f9a091a42cfef8542b Nov 03 00:45:28.648966 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:28.648860 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" event={"ID":"5e0059da-2321-4ed9-91f3-11af0d82efa7","Type":"ContainerStarted","Data":"7718769468f481ea5ceddcf0d70935dd4e5609e83bb25fcc1faa8135f5ac484c"} Nov 03 00:45:28.648966 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:28.648921 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" event={"ID":"5e0059da-2321-4ed9-91f3-11af0d82efa7","Type":"ContainerStarted","Data":"98d8c2fe29dbcdaadf8ec578d876b806adfe026a89a927f9a091a42cfef8542b"} Nov 03 00:45:28.757575 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:28.757514 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" podStartSLOduration=1.757499154 podStartE2EDuration="1.757499154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:28.755969805 +0000 UTC m=+6722.163281636" watchObservedRunningTime="2026-11-03 00:45:28.757499154 +0000 UTC m=+6722.164810980" Nov 03 00:45:29.655484 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:29.655441 5672 generic.go:356] "Generic (PLEG): container finished" podID="5e0059da-2321-4ed9-91f3-11af0d82efa7" containerID="7718769468f481ea5ceddcf0d70935dd4e5609e83bb25fcc1faa8135f5ac484c" exitCode=0 Nov 03 00:45:29.655484 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:29.655474 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" event={"ID":"5e0059da-2321-4ed9-91f3-11af0d82efa7","Type":"ContainerDied","Data":"7718769468f481ea5ceddcf0d70935dd4e5609e83bb25fcc1faa8135f5ac484c"} Nov 03 00:45:30.676525 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:30.676491 5672 scope.go:122] "RemoveContainer" containerID="49bc9e7c923c40a20f2c3507ecad8054030ead134af713ac060d643de65c4d36" Nov 03 00:45:30.797638 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:30.797613 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:30.985481 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:30.985439 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-xks-ca\") pod \"5e0059da-2321-4ed9-91f3-11af0d82efa7\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " Nov 03 00:45:30.985706 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:30.985518 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-openshift-ca\") pod \"5e0059da-2321-4ed9-91f3-11af0d82efa7\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " Nov 03 00:45:30.985706 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:30.985599 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0059da-2321-4ed9-91f3-11af0d82efa7-kube-api-access-szd4x\" (UniqueName: \"kubernetes.io/projected/5e0059da-2321-4ed9-91f3-11af0d82efa7-kube-api-access-szd4x\") pod \"5e0059da-2321-4ed9-91f3-11af0d82efa7\" (UID: \"5e0059da-2321-4ed9-91f3-11af0d82efa7\") " Nov 03 00:45:30.985888 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:30.985865 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-openshift-ca" pod "5e0059da-2321-4ed9-91f3-11af0d82efa7" (UID: "5e0059da-2321-4ed9-91f3-11af0d82efa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:30.987627 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:30.987601 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-xks-ca" pod "5e0059da-2321-4ed9-91f3-11af0d82efa7" (UID: "5e0059da-2321-4ed9-91f3-11af0d82efa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:45:30.987717 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:30.987680 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0059da-2321-4ed9-91f3-11af0d82efa7-kube-api-access-szd4x" pod "5e0059da-2321-4ed9-91f3-11af0d82efa7" (UID: "5e0059da-2321-4ed9-91f3-11af0d82efa7"). InnerVolumeSpecName "kube-api-access-szd4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:31.086885 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:31.086848 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:45:31.086885 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:31.086880 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0059da-2321-4ed9-91f3-11af0d82efa7-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:45:31.086885 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:31.086893 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szd4x\" (UniqueName: \"kubernetes.io/projected/5e0059da-2321-4ed9-91f3-11af0d82efa7-kube-api-access-szd4x\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:45:31.666571 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:31.666541 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" Nov 03 00:45:31.666766 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:31.666540 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635820af-dx2tl" event={"ID":"5e0059da-2321-4ed9-91f3-11af0d82efa7","Type":"ContainerDied","Data":"98d8c2fe29dbcdaadf8ec578d876b806adfe026a89a927f9a091a42cfef8542b"} Nov 03 00:45:31.666766 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:31.666670 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98d8c2fe29dbcdaadf8ec578d876b806adfe026a89a927f9a091a42cfef8542b" Nov 03 00:45:38.050245 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.050198 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c"] Nov 03 00:45:38.050675 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.050623 5672 cpu_manager.go:414] "containerMap: removing container" podUID="5e0059da-2321-4ed9-91f3-11af0d82efa7" containerName="revoke-keys" Nov 03 00:45:38.050675 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.050636 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0059da-2321-4ed9-91f3-11af0d82efa7" containerName="revoke-keys" Nov 03 00:45:38.050675 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.050653 5672 cpu_manager.go:414] "containerMap: removing container" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerName="cleanup" Nov 03 00:45:38.050675 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.050659 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerName="cleanup" Nov 03 00:45:38.050908 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.050748 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0059da-2321-4ed9-91f3-11af0d82efa7" containerName="revoke-keys" Nov 03 00:45:38.050908 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.050764 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d24ff4f-182b-4b9e-b374-a77621a01f5e" containerName="cleanup" Nov 03 00:45:38.056109 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.056080 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:38.106951 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.106918 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c"] Nov 03 00:45:38.150910 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.150875 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:45:38.150910 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.150900 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:45:38.151127 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.150959 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:45:38.151726 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.151707 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:45:38.155069 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.155048 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-193a5b22-6257c\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:38.155128 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.155081 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-193a5b22-6257c\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:38.155265 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.155249 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsjn\" (UniqueName: \"kubernetes.io/projected/a596c93a-ab6b-4816-863c-a2a43f4509ff-kube-api-access-nhsjn\") pod \"maas-api-revoke-sub-193a5b22-6257c\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:38.256294 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.256254 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-193a5b22-6257c\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:38.256294 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.256294 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-193a5b22-6257c\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:38.256570 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.256389 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsjn\" (UniqueName: \"kubernetes.io/projected/a596c93a-ab6b-4816-863c-a2a43f4509ff-kube-api-access-nhsjn\") pod \"maas-api-revoke-sub-193a5b22-6257c\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:38.256986 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.256960 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-193a5b22-6257c\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:38.258786 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.258764 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-193a5b22-6257c\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:38.310204 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.310097 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsjn\" (UniqueName: \"kubernetes.io/projected/a596c93a-ab6b-4816-863c-a2a43f4509ff-kube-api-access-nhsjn\") pod \"maas-api-revoke-sub-193a5b22-6257c\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:38.369046 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.369008 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:38.731678 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:38.731654 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c"] Nov 03 00:45:38.733903 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:45:38.733874 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda596c93a_ab6b_4816_863c_a2a43f4509ff.slice/crio-0e6fec5e3e85e7d98449d3ee1b754c8adb403a76b31d982aca4a59806b380f27 WatchSource:0}: Error finding container 0e6fec5e3e85e7d98449d3ee1b754c8adb403a76b31d982aca4a59806b380f27: Status 404 returned error can't find the container with id 0e6fec5e3e85e7d98449d3ee1b754c8adb403a76b31d982aca4a59806b380f27 Nov 03 00:45:39.707597 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:39.707564 5672 generic.go:356] "Generic (PLEG): container finished" podID="a596c93a-ab6b-4816-863c-a2a43f4509ff" containerID="d50e22cfb82b99d0f0d83c0730ed73b0c5b8801bd04c28a6620146905e5bb3f7" exitCode=0 Nov 03 00:45:39.707980 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:39.707648 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" event={"ID":"a596c93a-ab6b-4816-863c-a2a43f4509ff","Type":"ContainerDied","Data":"d50e22cfb82b99d0f0d83c0730ed73b0c5b8801bd04c28a6620146905e5bb3f7"} Nov 03 00:45:39.707980 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:39.707695 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" event={"ID":"a596c93a-ab6b-4816-863c-a2a43f4509ff","Type":"ContainerStarted","Data":"0e6fec5e3e85e7d98449d3ee1b754c8adb403a76b31d982aca4a59806b380f27"} Nov 03 00:45:40.020672 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:40.020642 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:45:40.874218 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:40.874182 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:40.980490 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:40.979794 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-openshift-ca\") pod \"a596c93a-ab6b-4816-863c-a2a43f4509ff\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " Nov 03 00:45:40.980490 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:40.979842 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a596c93a-ab6b-4816-863c-a2a43f4509ff-kube-api-access-nhsjn\" (UniqueName: \"kubernetes.io/projected/a596c93a-ab6b-4816-863c-a2a43f4509ff-kube-api-access-nhsjn\") pod \"a596c93a-ab6b-4816-863c-a2a43f4509ff\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " Nov 03 00:45:40.980490 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:40.979946 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-xks-ca\") pod \"a596c93a-ab6b-4816-863c-a2a43f4509ff\" (UID: \"a596c93a-ab6b-4816-863c-a2a43f4509ff\") " Nov 03 00:45:40.980490 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:40.980267 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-openshift-ca" pod "a596c93a-ab6b-4816-863c-a2a43f4509ff" (UID: "a596c93a-ab6b-4816-863c-a2a43f4509ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:40.984901 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:40.984846 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a596c93a-ab6b-4816-863c-a2a43f4509ff-kube-api-access-nhsjn" pod "a596c93a-ab6b-4816-863c-a2a43f4509ff" (UID: "a596c93a-ab6b-4816-863c-a2a43f4509ff"). InnerVolumeSpecName "kube-api-access-nhsjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:40.984901 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:40.983425 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-xks-ca" pod "a596c93a-ab6b-4816-863c-a2a43f4509ff" (UID: "a596c93a-ab6b-4816-863c-a2a43f4509ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:45:41.081757 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:41.081707 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:45:41.081757 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:41.081751 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a596c93a-ab6b-4816-863c-a2a43f4509ff-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:45:41.082024 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:41.081767 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhsjn\" (UniqueName: \"kubernetes.io/projected/a596c93a-ab6b-4816-863c-a2a43f4509ff-kube-api-access-nhsjn\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:45:41.720828 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:41.720797 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" Nov 03 00:45:41.721008 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:41.720791 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-193a5b22-6257c" event={"ID":"a596c93a-ab6b-4816-863c-a2a43f4509ff","Type":"ContainerDied","Data":"0e6fec5e3e85e7d98449d3ee1b754c8adb403a76b31d982aca4a59806b380f27"} Nov 03 00:45:41.721008 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:41.720926 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e6fec5e3e85e7d98449d3ee1b754c8adb403a76b31d982aca4a59806b380f27" Nov 03 00:45:50.745199 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:50.745159 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:45:52.733311 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:52.733266 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:45:52.733936 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:45:52.733901 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:45:56.020660 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:45:56.020616 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:46:00.457550 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:00.457511 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:46:22.847693 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:22.847614 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:46:26.088641 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:46:26.088565 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:46:29.027701 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.027666 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc"] Nov 03 00:46:29.028190 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.028092 5672 cpu_manager.go:414] "containerMap: removing container" podUID="a596c93a-ab6b-4816-863c-a2a43f4509ff" containerName="revoke-keys" Nov 03 00:46:29.028190 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.028105 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a596c93a-ab6b-4816-863c-a2a43f4509ff" containerName="revoke-keys" Nov 03 00:46:29.028274 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.028196 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="a596c93a-ab6b-4816-863c-a2a43f4509ff" containerName="revoke-keys" Nov 03 00:46:29.031476 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.031453 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:29.082406 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.082375 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:46:29.082888 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.082865 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:46:29.128630 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.128594 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:46:29.128768 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.128726 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:46:29.130670 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.130650 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc"] Nov 03 00:46:29.138038 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.138018 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zz7h\" (UniqueName: \"kubernetes.io/projected/55563851-8813-4b9d-8633-a5e53f8a8243-kube-api-access-4zz7h\") pod \"maas-api-revoke-sub-a4e0954b-m48wc\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:29.138115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.138063 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4e0954b-m48wc\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:29.138197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.138180 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4e0954b-m48wc\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:29.238773 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.238742 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4e0954b-m48wc\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:29.238914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.238785 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zz7h\" (UniqueName: \"kubernetes.io/projected/55563851-8813-4b9d-8633-a5e53f8a8243-kube-api-access-4zz7h\") pod \"maas-api-revoke-sub-a4e0954b-m48wc\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:29.238914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.238849 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4e0954b-m48wc\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:29.239558 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.239529 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4e0954b-m48wc\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:29.241249 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.241226 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4e0954b-m48wc\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:29.291929 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.291870 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zz7h\" (UniqueName: \"kubernetes.io/projected/55563851-8813-4b9d-8633-a5e53f8a8243-kube-api-access-4zz7h\") pod \"maas-api-revoke-sub-a4e0954b-m48wc\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:29.344971 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.344946 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:29.712485 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:29.712457 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc"] Nov 03 00:46:29.715018 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:46:29.714976 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55563851_8813_4b9d_8633_a5e53f8a8243.slice/crio-a4157fd591c173fa620e9266ab986ecf36dc4f552577ba260acdabda64fc4cc8 WatchSource:0}: Error finding container a4157fd591c173fa620e9266ab986ecf36dc4f552577ba260acdabda64fc4cc8: Status 404 returned error can't find the container with id a4157fd591c173fa620e9266ab986ecf36dc4f552577ba260acdabda64fc4cc8 Nov 03 00:46:30.020119 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:30.020083 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" event={"ID":"55563851-8813-4b9d-8633-a5e53f8a8243","Type":"ContainerStarted","Data":"29df83209e82ad02f6a520ed609ed884236ff95f96656a8edc530f881bd60422"} Nov 03 00:46:30.020277 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:30.020164 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" event={"ID":"55563851-8813-4b9d-8633-a5e53f8a8243","Type":"ContainerStarted","Data":"a4157fd591c173fa620e9266ab986ecf36dc4f552577ba260acdabda64fc4cc8"} Nov 03 00:46:30.129807 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:30.129751 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" podStartSLOduration=2.129737534 podStartE2EDuration="2.129737534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:30.127292506 +0000 UTC m=+6783.534604336" watchObservedRunningTime="2026-11-03 00:46:30.129737534 +0000 UTC m=+6783.537049364" Nov 03 00:46:31.026087 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:31.026054 5672 generic.go:356] "Generic (PLEG): container finished" podID="55563851-8813-4b9d-8633-a5e53f8a8243" containerID="29df83209e82ad02f6a520ed609ed884236ff95f96656a8edc530f881bd60422" exitCode=0 Nov 03 00:46:31.026292 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:31.026143 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" event={"ID":"55563851-8813-4b9d-8633-a5e53f8a8243","Type":"ContainerDied","Data":"29df83209e82ad02f6a520ed609ed884236ff95f96656a8edc530f881bd60422"} Nov 03 00:46:32.158301 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:32.158275 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:32.266034 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:32.265991 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55563851-8813-4b9d-8633-a5e53f8a8243-kube-api-access-4zz7h\" (UniqueName: \"kubernetes.io/projected/55563851-8813-4b9d-8633-a5e53f8a8243-kube-api-access-4zz7h\") pod \"55563851-8813-4b9d-8633-a5e53f8a8243\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " Nov 03 00:46:32.266238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:32.266092 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-xks-ca\") pod \"55563851-8813-4b9d-8633-a5e53f8a8243\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " Nov 03 00:46:32.266238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:32.266174 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-openshift-ca\") pod \"55563851-8813-4b9d-8633-a5e53f8a8243\" (UID: \"55563851-8813-4b9d-8633-a5e53f8a8243\") " Nov 03 00:46:32.266629 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:32.266582 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-openshift-ca" pod "55563851-8813-4b9d-8633-a5e53f8a8243" (UID: "55563851-8813-4b9d-8633-a5e53f8a8243"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:32.268857 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:32.268827 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55563851-8813-4b9d-8633-a5e53f8a8243-kube-api-access-4zz7h" pod "55563851-8813-4b9d-8633-a5e53f8a8243" (UID: "55563851-8813-4b9d-8633-a5e53f8a8243"). InnerVolumeSpecName "kube-api-access-4zz7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:32.269431 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:32.269403 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-xks-ca" pod "55563851-8813-4b9d-8633-a5e53f8a8243" (UID: "55563851-8813-4b9d-8633-a5e53f8a8243"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:32.367860 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:32.367788 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zz7h\" (UniqueName: \"kubernetes.io/projected/55563851-8813-4b9d-8633-a5e53f8a8243-kube-api-access-4zz7h\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:32.367860 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:32.367818 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:32.367860 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:32.367832 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55563851-8813-4b9d-8633-a5e53f8a8243-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:33.037093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:33.037006 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" Nov 03 00:46:33.037093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:33.037040 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0954b-m48wc" event={"ID":"55563851-8813-4b9d-8633-a5e53f8a8243","Type":"ContainerDied","Data":"a4157fd591c173fa620e9266ab986ecf36dc4f552577ba260acdabda64fc4cc8"} Nov 03 00:46:33.037093 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:33.037089 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4157fd591c173fa620e9266ab986ecf36dc4f552577ba260acdabda64fc4cc8" Nov 03 00:46:40.251859 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.251822 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh"] Nov 03 00:46:40.252379 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.252238 5672 cpu_manager.go:414] "containerMap: removing container" podUID="55563851-8813-4b9d-8633-a5e53f8a8243" containerName="revoke-keys" Nov 03 00:46:40.252379 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.252251 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55563851-8813-4b9d-8633-a5e53f8a8243" containerName="revoke-keys" Nov 03 00:46:40.252379 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.252321 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="55563851-8813-4b9d-8633-a5e53f8a8243" containerName="revoke-keys" Nov 03 00:46:40.255199 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.255174 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:40.306166 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.306108 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:46:40.341436 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.341404 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42r64\" (UniqueName: \"kubernetes.io/projected/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-kube-api-access-42r64\") pod \"maas-api-revoke-sub-0a896428-9phvh\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:40.341598 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.341458 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a896428-9phvh\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:40.341598 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.341589 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a896428-9phvh\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:40.352472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.352439 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:46:40.352472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.352459 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:46:40.352472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.352451 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:46:40.354612 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.354588 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh"] Nov 03 00:46:40.442437 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.442405 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a896428-9phvh\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:40.442560 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.442493 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a896428-9phvh\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:40.442560 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.442551 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42r64\" (UniqueName: \"kubernetes.io/projected/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-kube-api-access-42r64\") pod \"maas-api-revoke-sub-0a896428-9phvh\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:40.443133 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.443113 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a896428-9phvh\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:40.444761 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.444742 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a896428-9phvh\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:40.500266 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.500247 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42r64\" (UniqueName: \"kubernetes.io/projected/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-kube-api-access-42r64\") pod \"maas-api-revoke-sub-0a896428-9phvh\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:40.568176 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.568094 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:40.932795 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:40.932767 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh"] Nov 03 00:46:40.935503 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:46:40.935469 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6dc5fd_4368_45fd_ae37_3b07f3cf2c7c.slice/crio-6b0fa9f5404992efc394454697ccb18de2e617a0fc42fe7c5249ed93ecee5b46 WatchSource:0}: Error finding container 6b0fa9f5404992efc394454697ccb18de2e617a0fc42fe7c5249ed93ecee5b46: Status 404 returned error can't find the container with id 6b0fa9f5404992efc394454697ccb18de2e617a0fc42fe7c5249ed93ecee5b46 Nov 03 00:46:41.077139 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:41.077109 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" event={"ID":"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c","Type":"ContainerStarted","Data":"6b0fa9f5404992efc394454697ccb18de2e617a0fc42fe7c5249ed93ecee5b46"} Nov 03 00:46:42.083843 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:42.083810 5672 generic.go:356] "Generic (PLEG): container finished" podID="5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c" containerID="b012bd9be3ebe25d9897baaf3bef3639a837de6deadbf03b40647bf33652f11f" exitCode=0 Nov 03 00:46:42.084228 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:42.083899 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" event={"ID":"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c","Type":"ContainerDied","Data":"b012bd9be3ebe25d9897baaf3bef3639a837de6deadbf03b40647bf33652f11f"} Nov 03 00:46:43.217512 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:43.217484 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:43.268303 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:43.268278 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-xks-ca\") pod \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " Nov 03 00:46:43.268445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:43.268326 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-kube-api-access-42r64\" (UniqueName: \"kubernetes.io/projected/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-kube-api-access-42r64\") pod \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " Nov 03 00:46:43.268445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:43.268376 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-openshift-ca\") pod \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\" (UID: \"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c\") " Nov 03 00:46:43.268816 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:43.268789 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-openshift-ca" pod "5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c" (UID: "5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:43.270282 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:43.270262 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-xks-ca" pod "5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c" (UID: "5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:43.270529 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:43.270504 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-kube-api-access-42r64" pod "5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c" (UID: "5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c"). InnerVolumeSpecName "kube-api-access-42r64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:43.369640 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:43.369569 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42r64\" (UniqueName: \"kubernetes.io/projected/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-kube-api-access-42r64\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:43.369640 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:43.369597 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:43.369640 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:43.369608 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:43.530304 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:43.530273 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:46:44.097886 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:44.097842 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" event={"ID":"5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c","Type":"ContainerDied","Data":"6b0fa9f5404992efc394454697ccb18de2e617a0fc42fe7c5249ed93ecee5b46"} Nov 03 00:46:44.097886 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:44.097880 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a896428-9phvh" Nov 03 00:46:44.098219 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:44.097906 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b0fa9f5404992efc394454697ccb18de2e617a0fc42fe7c5249ed93ecee5b46" Nov 03 00:46:51.097067 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.097028 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx"] Nov 03 00:46:51.097616 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.097594 5672 cpu_manager.go:414] "containerMap: removing container" podUID="5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c" containerName="revoke-keys" Nov 03 00:46:51.097705 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.097618 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c" containerName="revoke-keys" Nov 03 00:46:51.097770 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.097744 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c" containerName="revoke-keys" Nov 03 00:46:51.100942 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.100917 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:51.149723 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.149693 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ns4\" (UniqueName: \"kubernetes.io/projected/c120bc2e-1ee7-40d5-b462-990eeee794bb-kube-api-access-56ns4\") pod \"maas-api-revoke-sub-d4d0c503-rrcvx\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:51.149911 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.149751 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d0c503-rrcvx\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:51.149911 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.149768 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d0c503-rrcvx\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:51.151226 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.151203 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:46:51.197367 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.197332 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:46:51.197529 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.197389 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:46:51.197529 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.197404 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:46:51.199545 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.199521 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx"] Nov 03 00:46:51.251070 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.251038 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ns4\" (UniqueName: \"kubernetes.io/projected/c120bc2e-1ee7-40d5-b462-990eeee794bb-kube-api-access-56ns4\") pod \"maas-api-revoke-sub-d4d0c503-rrcvx\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:51.251232 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.251111 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d0c503-rrcvx\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:51.251232 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.251138 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d0c503-rrcvx\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:51.251767 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.251744 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d0c503-rrcvx\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:51.253475 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.253451 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d0c503-rrcvx\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:51.304336 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.304308 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ns4\" (UniqueName: \"kubernetes.io/projected/c120bc2e-1ee7-40d5-b462-990eeee794bb-kube-api-access-56ns4\") pod \"maas-api-revoke-sub-d4d0c503-rrcvx\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:51.414073 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.414011 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:51.778617 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:51.778556 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx"] Nov 03 00:46:51.781039 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:46:51.781006 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc120bc2e_1ee7_40d5_b462_990eeee794bb.slice/crio-8757bfe9a6a0fc0e61cc88980a93b62016033833786f278a4ddfbd7177345dd2 WatchSource:0}: Error finding container 8757bfe9a6a0fc0e61cc88980a93b62016033833786f278a4ddfbd7177345dd2: Status 404 returned error can't find the container with id 8757bfe9a6a0fc0e61cc88980a93b62016033833786f278a4ddfbd7177345dd2 Nov 03 00:46:52.146492 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:52.146456 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" event={"ID":"c120bc2e-1ee7-40d5-b462-990eeee794bb","Type":"ContainerStarted","Data":"cd27db7df5e702601a756448c10efe310bdc951f8418de59cf197172e340f46d"} Nov 03 00:46:52.146946 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:52.146506 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" event={"ID":"c120bc2e-1ee7-40d5-b462-990eeee794bb","Type":"ContainerStarted","Data":"8757bfe9a6a0fc0e61cc88980a93b62016033833786f278a4ddfbd7177345dd2"} Nov 03 00:46:52.254329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:52.254283 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" podStartSLOduration=1.254269976 podStartE2EDuration="1.254269976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:52.252368072 +0000 UTC m=+6805.659679901" watchObservedRunningTime="2026-11-03 00:46:52.254269976 +0000 UTC m=+6805.661581796" Nov 03 00:46:53.153843 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:53.153812 5672 generic.go:356] "Generic (PLEG): container finished" podID="c120bc2e-1ee7-40d5-b462-990eeee794bb" containerID="cd27db7df5e702601a756448c10efe310bdc951f8418de59cf197172e340f46d" exitCode=0 Nov 03 00:46:53.154293 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:53.153894 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" event={"ID":"c120bc2e-1ee7-40d5-b462-990eeee794bb","Type":"ContainerDied","Data":"cd27db7df5e702601a756448c10efe310bdc951f8418de59cf197172e340f46d"} Nov 03 00:46:54.287181 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:54.287138 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:54.380188 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:54.380141 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-openshift-ca\") pod \"c120bc2e-1ee7-40d5-b462-990eeee794bb\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " Nov 03 00:46:54.380341 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:54.380227 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c120bc2e-1ee7-40d5-b462-990eeee794bb-kube-api-access-56ns4\" (UniqueName: \"kubernetes.io/projected/c120bc2e-1ee7-40d5-b462-990eeee794bb-kube-api-access-56ns4\") pod \"c120bc2e-1ee7-40d5-b462-990eeee794bb\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " Nov 03 00:46:54.380341 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:54.380260 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-xks-ca\") pod \"c120bc2e-1ee7-40d5-b462-990eeee794bb\" (UID: \"c120bc2e-1ee7-40d5-b462-990eeee794bb\") " Nov 03 00:46:54.380555 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:54.380530 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-openshift-ca" pod "c120bc2e-1ee7-40d5-b462-990eeee794bb" (UID: "c120bc2e-1ee7-40d5-b462-990eeee794bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:54.382315 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:54.382285 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-xks-ca" pod "c120bc2e-1ee7-40d5-b462-990eeee794bb" (UID: "c120bc2e-1ee7-40d5-b462-990eeee794bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:54.382445 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:54.382333 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c120bc2e-1ee7-40d5-b462-990eeee794bb-kube-api-access-56ns4" pod "c120bc2e-1ee7-40d5-b462-990eeee794bb" (UID: "c120bc2e-1ee7-40d5-b462-990eeee794bb"). InnerVolumeSpecName "kube-api-access-56ns4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:54.481701 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:54.481675 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:54.481701 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:54.481699 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56ns4\" (UniqueName: \"kubernetes.io/projected/c120bc2e-1ee7-40d5-b462-990eeee794bb-kube-api-access-56ns4\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:54.481897 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:54.481709 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c120bc2e-1ee7-40d5-b462-990eeee794bb-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:55.164787 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:55.164756 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" Nov 03 00:46:55.164967 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:55.164793 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d0c503-rrcvx" event={"ID":"c120bc2e-1ee7-40d5-b462-990eeee794bb","Type":"ContainerDied","Data":"8757bfe9a6a0fc0e61cc88980a93b62016033833786f278a4ddfbd7177345dd2"} Nov 03 00:46:55.164967 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:46:55.164835 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8757bfe9a6a0fc0e61cc88980a93b62016033833786f278a4ddfbd7177345dd2" Nov 03 00:46:56.019521 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:46:56.019479 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:47:02.364119 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.364076 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p"] Nov 03 00:47:02.364697 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.364677 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c120bc2e-1ee7-40d5-b462-990eeee794bb" containerName="revoke-keys" Nov 03 00:47:02.364746 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.364701 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c120bc2e-1ee7-40d5-b462-990eeee794bb" containerName="revoke-keys" Nov 03 00:47:02.364821 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.364810 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c120bc2e-1ee7-40d5-b462-990eeee794bb" containerName="revoke-keys" Nov 03 00:47:02.368431 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.368405 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:02.420465 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.420433 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p"] Nov 03 00:47:02.453853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.453825 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150516ef-pdh6p\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:02.454001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.453917 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150516ef-pdh6p\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:02.454001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.453970 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7kh\" (UniqueName: \"kubernetes.io/projected/108292e2-a977-4b33-a41b-28ebecedaef7-kube-api-access-gt7kh\") pod \"maas-api-revoke-sub-150516ef-pdh6p\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:02.463899 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.463869 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:47:02.463899 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.463886 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:47:02.463899 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.463886 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:47:02.464113 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.463916 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:47:02.554692 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.554660 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150516ef-pdh6p\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:02.554841 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.554719 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150516ef-pdh6p\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:02.554841 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.554787 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7kh\" (UniqueName: \"kubernetes.io/projected/108292e2-a977-4b33-a41b-28ebecedaef7-kube-api-access-gt7kh\") pod \"maas-api-revoke-sub-150516ef-pdh6p\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:02.555422 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.555404 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150516ef-pdh6p\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:02.557084 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.557062 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150516ef-pdh6p\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:02.608240 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.608218 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7kh\" (UniqueName: \"kubernetes.io/projected/108292e2-a977-4b33-a41b-28ebecedaef7-kube-api-access-gt7kh\") pod \"maas-api-revoke-sub-150516ef-pdh6p\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:02.685488 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:02.685405 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:03.049221 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:03.049196 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p"] Nov 03 00:47:03.050949 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:47:03.050921 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108292e2_a977_4b33_a41b_28ebecedaef7.slice/crio-294707212de2ce340af72f74039751245bcfae73ad2ea2741e0193fb5494b0b4 WatchSource:0}: Error finding container 294707212de2ce340af72f74039751245bcfae73ad2ea2741e0193fb5494b0b4: Status 404 returned error can't find the container with id 294707212de2ce340af72f74039751245bcfae73ad2ea2741e0193fb5494b0b4 Nov 03 00:47:03.212374 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:03.212336 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" event={"ID":"108292e2-a977-4b33-a41b-28ebecedaef7","Type":"ContainerStarted","Data":"294707212de2ce340af72f74039751245bcfae73ad2ea2741e0193fb5494b0b4"} Nov 03 00:47:04.218732 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:04.218702 5672 generic.go:356] "Generic (PLEG): container finished" podID="108292e2-a977-4b33-a41b-28ebecedaef7" containerID="9ae0900090a4979dc130116de52a845355acaa69c8995df8c1c6fd9ba528f9b8" exitCode=0 Nov 03 00:47:04.219103 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:04.218784 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" event={"ID":"108292e2-a977-4b33-a41b-28ebecedaef7","Type":"ContainerDied","Data":"9ae0900090a4979dc130116de52a845355acaa69c8995df8c1c6fd9ba528f9b8"} Nov 03 00:47:05.354495 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:05.354472 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:05.481478 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:05.481406 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-xks-ca\") pod \"108292e2-a977-4b33-a41b-28ebecedaef7\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " Nov 03 00:47:05.481478 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:05.481471 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-openshift-ca\") pod \"108292e2-a977-4b33-a41b-28ebecedaef7\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " Nov 03 00:47:05.481674 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:05.481563 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108292e2-a977-4b33-a41b-28ebecedaef7-kube-api-access-gt7kh\" (UniqueName: \"kubernetes.io/projected/108292e2-a977-4b33-a41b-28ebecedaef7-kube-api-access-gt7kh\") pod \"108292e2-a977-4b33-a41b-28ebecedaef7\" (UID: \"108292e2-a977-4b33-a41b-28ebecedaef7\") " Nov 03 00:47:05.481847 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:05.481824 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-openshift-ca" pod "108292e2-a977-4b33-a41b-28ebecedaef7" (UID: "108292e2-a977-4b33-a41b-28ebecedaef7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:05.481914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:05.481900 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:05.483601 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:05.483575 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108292e2-a977-4b33-a41b-28ebecedaef7-kube-api-access-gt7kh" pod "108292e2-a977-4b33-a41b-28ebecedaef7" (UID: "108292e2-a977-4b33-a41b-28ebecedaef7"). InnerVolumeSpecName "kube-api-access-gt7kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:05.483729 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:05.483641 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-xks-ca" pod "108292e2-a977-4b33-a41b-28ebecedaef7" (UID: "108292e2-a977-4b33-a41b-28ebecedaef7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:05.582604 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:05.582572 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108292e2-a977-4b33-a41b-28ebecedaef7-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:05.582604 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:05.582603 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt7kh\" (UniqueName: \"kubernetes.io/projected/108292e2-a977-4b33-a41b-28ebecedaef7-kube-api-access-gt7kh\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:06.233533 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:06.233495 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" Nov 03 00:47:06.233750 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:06.233585 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150516ef-pdh6p" event={"ID":"108292e2-a977-4b33-a41b-28ebecedaef7","Type":"ContainerDied","Data":"294707212de2ce340af72f74039751245bcfae73ad2ea2741e0193fb5494b0b4"} Nov 03 00:47:06.233750 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:06.233640 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="294707212de2ce340af72f74039751245bcfae73ad2ea2741e0193fb5494b0b4" Nov 03 00:47:13.109341 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.109302 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh"] Nov 03 00:47:13.109747 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.109668 5672 cpu_manager.go:414] "containerMap: removing container" podUID="108292e2-a977-4b33-a41b-28ebecedaef7" containerName="revoke-keys" Nov 03 00:47:13.109747 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.109679 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108292e2-a977-4b33-a41b-28ebecedaef7" containerName="revoke-keys" Nov 03 00:47:13.109827 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.109784 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="108292e2-a977-4b33-a41b-28ebecedaef7" containerName="revoke-keys" Nov 03 00:47:13.113128 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.113101 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:13.166580 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.166535 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685a8fcb-sk5bh\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:13.166798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.166620 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfsq2\" (UniqueName: \"kubernetes.io/projected/194aec71-fa21-4c8f-b595-5fff61d68732-kube-api-access-tfsq2\") pod \"maas-api-revoke-sub-685a8fcb-sk5bh\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:13.166798 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.166713 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685a8fcb-sk5bh\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:13.210207 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.210176 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:47:13.210207 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.210195 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:47:13.210433 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.210212 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:47:13.210433 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.210267 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:47:13.212495 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.212468 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh"] Nov 03 00:47:13.268115 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.268091 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685a8fcb-sk5bh\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:13.268286 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.268135 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfsq2\" (UniqueName: \"kubernetes.io/projected/194aec71-fa21-4c8f-b595-5fff61d68732-kube-api-access-tfsq2\") pod \"maas-api-revoke-sub-685a8fcb-sk5bh\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:13.268286 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.268194 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685a8fcb-sk5bh\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:13.268822 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.268796 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685a8fcb-sk5bh\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:13.270409 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.270386 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685a8fcb-sk5bh\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:13.321615 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.321589 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfsq2\" (UniqueName: \"kubernetes.io/projected/194aec71-fa21-4c8f-b595-5fff61d68732-kube-api-access-tfsq2\") pod \"maas-api-revoke-sub-685a8fcb-sk5bh\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:13.425196 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.425105 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:13.786557 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:13.786532 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh"] Nov 03 00:47:13.788588 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:47:13.788560 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194aec71_fa21_4c8f_b595_5fff61d68732.slice/crio-aa0b5f672fe1659c928c5beb693989fe95e950003e395bd017759595464a8a94 WatchSource:0}: Error finding container aa0b5f672fe1659c928c5beb693989fe95e950003e395bd017759595464a8a94: Status 404 returned error can't find the container with id aa0b5f672fe1659c928c5beb693989fe95e950003e395bd017759595464a8a94 Nov 03 00:47:14.337127 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:14.337091 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" event={"ID":"194aec71-fa21-4c8f-b595-5fff61d68732","Type":"ContainerStarted","Data":"93dfc949a67a65791a81e7cfc75dc456b26c5eb277bc6e58147c55246e37f82e"} Nov 03 00:47:14.337625 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:14.337164 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" event={"ID":"194aec71-fa21-4c8f-b595-5fff61d68732","Type":"ContainerStarted","Data":"aa0b5f672fe1659c928c5beb693989fe95e950003e395bd017759595464a8a94"} Nov 03 00:47:14.443951 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:14.443898 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" podStartSLOduration=1.443883201 podStartE2EDuration="1.443883201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:14.442968004 +0000 UTC m=+6827.850279846" watchObservedRunningTime="2026-11-03 00:47:14.443883201 +0000 UTC m=+6827.851195054" Nov 03 00:47:15.344100 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:15.344062 5672 generic.go:356] "Generic (PLEG): container finished" podID="194aec71-fa21-4c8f-b595-5fff61d68732" containerID="93dfc949a67a65791a81e7cfc75dc456b26c5eb277bc6e58147c55246e37f82e" exitCode=0 Nov 03 00:47:15.344527 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:15.344172 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" event={"ID":"194aec71-fa21-4c8f-b595-5fff61d68732","Type":"ContainerDied","Data":"93dfc949a67a65791a81e7cfc75dc456b26c5eb277bc6e58147c55246e37f82e"} Nov 03 00:47:16.477082 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:16.477045 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:16.596425 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:16.596399 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-openshift-ca\") pod \"194aec71-fa21-4c8f-b595-5fff61d68732\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " Nov 03 00:47:16.596584 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:16.596442 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194aec71-fa21-4c8f-b595-5fff61d68732-kube-api-access-tfsq2\" (UniqueName: \"kubernetes.io/projected/194aec71-fa21-4c8f-b595-5fff61d68732-kube-api-access-tfsq2\") pod \"194aec71-fa21-4c8f-b595-5fff61d68732\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " Nov 03 00:47:16.596584 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:16.596507 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-xks-ca\") pod \"194aec71-fa21-4c8f-b595-5fff61d68732\" (UID: \"194aec71-fa21-4c8f-b595-5fff61d68732\") " Nov 03 00:47:16.596859 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:16.596833 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-openshift-ca" pod "194aec71-fa21-4c8f-b595-5fff61d68732" (UID: "194aec71-fa21-4c8f-b595-5fff61d68732"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:16.598561 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:16.598529 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194aec71-fa21-4c8f-b595-5fff61d68732-kube-api-access-tfsq2" pod "194aec71-fa21-4c8f-b595-5fff61d68732" (UID: "194aec71-fa21-4c8f-b595-5fff61d68732"). InnerVolumeSpecName "kube-api-access-tfsq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:16.598561 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:16.598546 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-xks-ca" pod "194aec71-fa21-4c8f-b595-5fff61d68732" (UID: "194aec71-fa21-4c8f-b595-5fff61d68732"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:16.697534 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:16.697498 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:16.697534 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:16.697526 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfsq2\" (UniqueName: \"kubernetes.io/projected/194aec71-fa21-4c8f-b595-5fff61d68732-kube-api-access-tfsq2\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:16.697534 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:16.697539 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/194aec71-fa21-4c8f-b595-5fff61d68732-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:17.356630 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:17.356599 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" Nov 03 00:47:17.356809 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:17.356593 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685a8fcb-sk5bh" event={"ID":"194aec71-fa21-4c8f-b595-5fff61d68732","Type":"ContainerDied","Data":"aa0b5f672fe1659c928c5beb693989fe95e950003e395bd017759595464a8a94"} Nov 03 00:47:17.356809 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:17.356749 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa0b5f672fe1659c928c5beb693989fe95e950003e395bd017759595464a8a94" Nov 03 00:47:24.236812 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.236773 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb"] Nov 03 00:47:24.237244 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.237184 5672 cpu_manager.go:414] "containerMap: removing container" podUID="194aec71-fa21-4c8f-b595-5fff61d68732" containerName="revoke-keys" Nov 03 00:47:24.237244 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.237200 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194aec71-fa21-4c8f-b595-5fff61d68732" containerName="revoke-keys" Nov 03 00:47:24.237347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.237289 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="194aec71-fa21-4c8f-b595-5fff61d68732" containerName="revoke-keys" Nov 03 00:47:24.240191 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.240169 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:24.293349 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.293315 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb"] Nov 03 00:47:24.336900 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.336853 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:47:24.336900 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.336874 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:47:24.337207 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.337025 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:47:24.337366 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.337340 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:47:24.364785 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.364750 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f15af6b0-ss9gb\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:24.364904 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.364825 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f15af6b0-ss9gb\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:24.364904 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.364859 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prjg4\" (UniqueName: \"kubernetes.io/projected/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-kube-api-access-prjg4\") pod \"maas-api-revoke-sub-f15af6b0-ss9gb\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:24.465414 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.465383 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f15af6b0-ss9gb\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:24.465560 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.465434 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f15af6b0-ss9gb\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:24.465560 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.465454 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prjg4\" (UniqueName: \"kubernetes.io/projected/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-kube-api-access-prjg4\") pod \"maas-api-revoke-sub-f15af6b0-ss9gb\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:24.466012 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.465992 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f15af6b0-ss9gb\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:24.467711 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.467687 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f15af6b0-ss9gb\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:24.518896 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.518832 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prjg4\" (UniqueName: \"kubernetes.io/projected/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-kube-api-access-prjg4\") pod \"maas-api-revoke-sub-f15af6b0-ss9gb\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:24.552842 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.552820 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:24.916353 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:24.916319 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb"] Nov 03 00:47:24.916731 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:47:24.916709 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03e08d3_62f6_4fd0_b0c9_7e62d84c4a70.slice/crio-c69ffd98b509fe88595eee6d371351805121ed8e0b2ef5ad943393b38a73136f WatchSource:0}: Error finding container c69ffd98b509fe88595eee6d371351805121ed8e0b2ef5ad943393b38a73136f: Status 404 returned error can't find the container with id c69ffd98b509fe88595eee6d371351805121ed8e0b2ef5ad943393b38a73136f Nov 03 00:47:25.400203 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:25.400162 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" event={"ID":"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70","Type":"ContainerStarted","Data":"3195d0bf4445d159896f1bf35a12715b8fc62b22df908824a6a6a4385853cb4f"} Nov 03 00:47:25.400684 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:25.400226 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" event={"ID":"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70","Type":"ContainerStarted","Data":"c69ffd98b509fe88595eee6d371351805121ed8e0b2ef5ad943393b38a73136f"} Nov 03 00:47:25.508301 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:25.508251 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" podStartSLOduration=1.508236911 podStartE2EDuration="1.508236911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:25.506315429 +0000 UTC m=+6838.913627259" watchObservedRunningTime="2026-11-03 00:47:25.508236911 +0000 UTC m=+6838.915548742" Nov 03 00:47:26.020185 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:47:26.020134 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:47:26.408569 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:26.408535 5672 generic.go:356] "Generic (PLEG): container finished" podID="f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70" containerID="3195d0bf4445d159896f1bf35a12715b8fc62b22df908824a6a6a4385853cb4f" exitCode=0 Nov 03 00:47:26.408991 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:26.408625 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" event={"ID":"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70","Type":"ContainerDied","Data":"3195d0bf4445d159896f1bf35a12715b8fc62b22df908824a6a6a4385853cb4f"} Nov 03 00:47:27.543950 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:27.543923 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:27.696127 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:27.696052 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-xks-ca\") pod \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " Nov 03 00:47:27.696297 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:27.696211 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-kube-api-access-prjg4\" (UniqueName: \"kubernetes.io/projected/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-kube-api-access-prjg4\") pod \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " Nov 03 00:47:27.696297 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:27.696253 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-openshift-ca\") pod \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\" (UID: \"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70\") " Nov 03 00:47:27.696627 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:27.696602 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-openshift-ca" pod "f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70" (UID: "f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:27.698098 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:27.698071 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-xks-ca" pod "f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70" (UID: "f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:27.698215 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:27.698107 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-kube-api-access-prjg4" pod "f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70" (UID: "f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70"). InnerVolumeSpecName "kube-api-access-prjg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:27.797620 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:27.797586 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prjg4\" (UniqueName: \"kubernetes.io/projected/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-kube-api-access-prjg4\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:27.797620 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:27.797615 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:27.797851 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:27.797630 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:28.445169 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:28.438909 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" event={"ID":"f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70","Type":"ContainerDied","Data":"c69ffd98b509fe88595eee6d371351805121ed8e0b2ef5ad943393b38a73136f"} Nov 03 00:47:28.445169 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:28.439007 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c69ffd98b509fe88595eee6d371351805121ed8e0b2ef5ad943393b38a73136f" Nov 03 00:47:28.445169 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:28.439098 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f15af6b0-ss9gb" Nov 03 00:47:34.943800 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:34.943762 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56"] Nov 03 00:47:34.944354 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:34.944333 5672 cpu_manager.go:414] "containerMap: removing container" podUID="f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70" containerName="revoke-keys" Nov 03 00:47:34.944440 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:34.944356 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70" containerName="revoke-keys" Nov 03 00:47:34.944536 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:34.944523 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70" containerName="revoke-keys" Nov 03 00:47:34.948214 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:34.948191 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:35.000302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.000268 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56"] Nov 03 00:47:35.043878 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.043840 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:47:35.044062 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.043972 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:47:35.044062 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.043976 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:47:35.044062 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.043974 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:47:35.065100 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.065064 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcc24\" (UniqueName: \"kubernetes.io/projected/fc71ebed-03c4-4e8f-809a-527518a10a51-kube-api-access-qcc24\") pod \"maas-api-revoke-sub-a0955be3-tbr56\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:35.065264 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.065105 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0955be3-tbr56\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:35.065264 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.065132 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0955be3-tbr56\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:35.166083 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.166046 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcc24\" (UniqueName: \"kubernetes.io/projected/fc71ebed-03c4-4e8f-809a-527518a10a51-kube-api-access-qcc24\") pod \"maas-api-revoke-sub-a0955be3-tbr56\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:35.166083 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.166084 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0955be3-tbr56\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:35.166376 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.166107 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0955be3-tbr56\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:35.166774 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.166752 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0955be3-tbr56\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:35.168461 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.168435 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0955be3-tbr56\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:35.219843 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.219811 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcc24\" (UniqueName: \"kubernetes.io/projected/fc71ebed-03c4-4e8f-809a-527518a10a51-kube-api-access-qcc24\") pod \"maas-api-revoke-sub-a0955be3-tbr56\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:35.260394 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.260362 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:35.627728 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:35.627689 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56"] Nov 03 00:47:35.633622 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:47:35.633594 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc71ebed_03c4_4e8f_809a_527518a10a51.slice/crio-d39cc1056169b65cacceff2222401fe26f632ed28fb112421222eeb027a68e74 WatchSource:0}: Error finding container d39cc1056169b65cacceff2222401fe26f632ed28fb112421222eeb027a68e74: Status 404 returned error can't find the container with id d39cc1056169b65cacceff2222401fe26f632ed28fb112421222eeb027a68e74 Nov 03 00:47:36.516874 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:36.516836 5672 generic.go:356] "Generic (PLEG): container finished" podID="fc71ebed-03c4-4e8f-809a-527518a10a51" containerID="16d24717af19a4cf1a3ec87c5486877e91e09305687a11222e33fd5a840806cd" exitCode=0 Nov 03 00:47:36.517329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:36.516880 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" event={"ID":"fc71ebed-03c4-4e8f-809a-527518a10a51","Type":"ContainerDied","Data":"16d24717af19a4cf1a3ec87c5486877e91e09305687a11222e33fd5a840806cd"} Nov 03 00:47:36.517329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:36.516952 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" event={"ID":"fc71ebed-03c4-4e8f-809a-527518a10a51","Type":"ContainerStarted","Data":"d39cc1056169b65cacceff2222401fe26f632ed28fb112421222eeb027a68e74"} Nov 03 00:47:37.653989 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:37.653962 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:37.790900 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:37.790825 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-xks-ca\") pod \"fc71ebed-03c4-4e8f-809a-527518a10a51\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " Nov 03 00:47:37.790900 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:37.790875 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc71ebed-03c4-4e8f-809a-527518a10a51-kube-api-access-qcc24\" (UniqueName: \"kubernetes.io/projected/fc71ebed-03c4-4e8f-809a-527518a10a51-kube-api-access-qcc24\") pod \"fc71ebed-03c4-4e8f-809a-527518a10a51\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " Nov 03 00:47:37.791096 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:37.791057 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-openshift-ca\") pod \"fc71ebed-03c4-4e8f-809a-527518a10a51\" (UID: \"fc71ebed-03c4-4e8f-809a-527518a10a51\") " Nov 03 00:47:37.791421 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:37.791396 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-openshift-ca" pod "fc71ebed-03c4-4e8f-809a-527518a10a51" (UID: "fc71ebed-03c4-4e8f-809a-527518a10a51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:37.792865 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:37.792844 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-xks-ca" pod "fc71ebed-03c4-4e8f-809a-527518a10a51" (UID: "fc71ebed-03c4-4e8f-809a-527518a10a51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:37.792949 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:37.792922 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc71ebed-03c4-4e8f-809a-527518a10a51-kube-api-access-qcc24" pod "fc71ebed-03c4-4e8f-809a-527518a10a51" (UID: "fc71ebed-03c4-4e8f-809a-527518a10a51"). InnerVolumeSpecName "kube-api-access-qcc24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:37.892675 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:37.892650 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:37.892675 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:37.892676 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc71ebed-03c4-4e8f-809a-527518a10a51-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:37.892824 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:37.892685 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcc24\" (UniqueName: \"kubernetes.io/projected/fc71ebed-03c4-4e8f-809a-527518a10a51-kube-api-access-qcc24\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:38.527980 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:38.527951 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" Nov 03 00:47:38.527980 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:38.527965 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0955be3-tbr56" event={"ID":"fc71ebed-03c4-4e8f-809a-527518a10a51","Type":"ContainerDied","Data":"d39cc1056169b65cacceff2222401fe26f632ed28fb112421222eeb027a68e74"} Nov 03 00:47:38.528226 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:38.528007 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d39cc1056169b65cacceff2222401fe26f632ed28fb112421222eeb027a68e74" Nov 03 00:47:45.787306 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.787270 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l"] Nov 03 00:47:45.787704 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.787672 5672 cpu_manager.go:414] "containerMap: removing container" podUID="fc71ebed-03c4-4e8f-809a-527518a10a51" containerName="revoke-keys" Nov 03 00:47:45.787704 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.787682 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc71ebed-03c4-4e8f-809a-527518a10a51" containerName="revoke-keys" Nov 03 00:47:45.787777 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.787743 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc71ebed-03c4-4e8f-809a-527518a10a51" containerName="revoke-keys" Nov 03 00:47:45.790935 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.790913 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:45.886929 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.886895 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:47:45.886929 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.886897 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:47:45.887143 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.886937 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:47:45.887143 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.887001 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:47:45.889197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.889178 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l"] Nov 03 00:47:45.958290 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.958259 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1f8efb8-t297l\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:45.958466 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.958317 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6gkf\" (UniqueName: \"kubernetes.io/projected/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-kube-api-access-j6gkf\") pod \"maas-api-revoke-sub-e1f8efb8-t297l\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:45.958466 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:45.958423 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1f8efb8-t297l\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:46.059683 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:46.059596 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6gkf\" (UniqueName: \"kubernetes.io/projected/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-kube-api-access-j6gkf\") pod \"maas-api-revoke-sub-e1f8efb8-t297l\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:46.059683 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:46.059642 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1f8efb8-t297l\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:46.059888 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:46.059707 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1f8efb8-t297l\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:46.060417 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:46.060389 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1f8efb8-t297l\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:46.062022 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:46.062001 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1f8efb8-t297l\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:46.113108 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:46.113070 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6gkf\" (UniqueName: \"kubernetes.io/projected/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-kube-api-access-j6gkf\") pod \"maas-api-revoke-sub-e1f8efb8-t297l\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:46.404084 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:46.404009 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:46.766827 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:46.766797 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l"] Nov 03 00:47:46.768181 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:47:46.768131 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e72fd2_07f7_4d18_8a6c_c54f4eae16af.slice/crio-92a07759766eb2c6678f9ef626f15cd046634c67a5caf71447e35a70aaf199bc WatchSource:0}: Error finding container 92a07759766eb2c6678f9ef626f15cd046634c67a5caf71447e35a70aaf199bc: Status 404 returned error can't find the container with id 92a07759766eb2c6678f9ef626f15cd046634c67a5caf71447e35a70aaf199bc Nov 03 00:47:47.575969 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:47.575929 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" event={"ID":"55e72fd2-07f7-4d18-8a6c-c54f4eae16af","Type":"ContainerStarted","Data":"45c5fc5d063ffafad0aed152ffa2697caebe44db05106f57d524d0f8e2a4b268"} Nov 03 00:47:47.576470 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:47.575981 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" event={"ID":"55e72fd2-07f7-4d18-8a6c-c54f4eae16af","Type":"ContainerStarted","Data":"92a07759766eb2c6678f9ef626f15cd046634c67a5caf71447e35a70aaf199bc"} Nov 03 00:47:47.682955 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:47.682908 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" podStartSLOduration=2.6828916 podStartE2EDuration="2.6828916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:47.681484605 +0000 UTC m=+6861.088796434" watchObservedRunningTime="2026-11-03 00:47:47.6828916 +0000 UTC m=+6861.090203430" Nov 03 00:47:48.583546 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:48.583512 5672 generic.go:356] "Generic (PLEG): container finished" podID="55e72fd2-07f7-4d18-8a6c-c54f4eae16af" containerID="45c5fc5d063ffafad0aed152ffa2697caebe44db05106f57d524d0f8e2a4b268" exitCode=0 Nov 03 00:47:48.584096 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:48.583613 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" event={"ID":"55e72fd2-07f7-4d18-8a6c-c54f4eae16af","Type":"ContainerDied","Data":"45c5fc5d063ffafad0aed152ffa2697caebe44db05106f57d524d0f8e2a4b268"} Nov 03 00:47:49.995174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:49.995091 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:50.115208 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.115174 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-xks-ca\") pod \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " Nov 03 00:47:50.116302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.115526 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-kube-api-access-j6gkf\" (UniqueName: \"kubernetes.io/projected/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-kube-api-access-j6gkf\") pod \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " Nov 03 00:47:50.116302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.115668 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-openshift-ca\") pod \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\" (UID: \"55e72fd2-07f7-4d18-8a6c-c54f4eae16af\") " Nov 03 00:47:50.116302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.116057 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-openshift-ca" pod "55e72fd2-07f7-4d18-8a6c-c54f4eae16af" (UID: "55e72fd2-07f7-4d18-8a6c-c54f4eae16af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:50.117671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.117631 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-xks-ca" pod "55e72fd2-07f7-4d18-8a6c-c54f4eae16af" (UID: "55e72fd2-07f7-4d18-8a6c-c54f4eae16af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:50.118007 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.117980 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-kube-api-access-j6gkf" pod "55e72fd2-07f7-4d18-8a6c-c54f4eae16af" (UID: "55e72fd2-07f7-4d18-8a6c-c54f4eae16af"). InnerVolumeSpecName "kube-api-access-j6gkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:50.216806 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.216726 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:50.216806 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.216755 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:50.216806 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.216764 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6gkf\" (UniqueName: \"kubernetes.io/projected/55e72fd2-07f7-4d18-8a6c-c54f4eae16af-kube-api-access-j6gkf\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:50.612396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.612353 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" event={"ID":"55e72fd2-07f7-4d18-8a6c-c54f4eae16af","Type":"ContainerDied","Data":"92a07759766eb2c6678f9ef626f15cd046634c67a5caf71447e35a70aaf199bc"} Nov 03 00:47:50.612627 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.612421 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92a07759766eb2c6678f9ef626f15cd046634c67a5caf71447e35a70aaf199bc" Nov 03 00:47:50.612627 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:50.612506 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1f8efb8-t297l" Nov 03 00:47:56.020276 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:47:56.020237 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:47:56.566002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.565969 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk"] Nov 03 00:47:56.566418 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.566402 5672 cpu_manager.go:414] "containerMap: removing container" podUID="55e72fd2-07f7-4d18-8a6c-c54f4eae16af" containerName="revoke-keys" Nov 03 00:47:56.566472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.566419 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e72fd2-07f7-4d18-8a6c-c54f4eae16af" containerName="revoke-keys" Nov 03 00:47:56.566514 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.566506 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e72fd2-07f7-4d18-8a6c-c54f4eae16af" containerName="revoke-keys" Nov 03 00:47:56.569767 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.569745 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:56.571003 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.570979 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2feff40-sqrgk\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:56.571169 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.571125 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj8zk\" (UniqueName: \"kubernetes.io/projected/fbf8d220-bb7a-44f0-ae59-e110ace82b94-kube-api-access-qj8zk\") pod \"maas-api-revoke-sub-a2feff40-sqrgk\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:56.571266 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.571247 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2feff40-sqrgk\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:56.621167 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.621122 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:47:56.667399 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.667370 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:47:56.667399 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.667387 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:47:56.667610 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.667414 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:47:56.669601 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.669579 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk"] Nov 03 00:47:56.671940 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.671925 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj8zk\" (UniqueName: \"kubernetes.io/projected/fbf8d220-bb7a-44f0-ae59-e110ace82b94-kube-api-access-qj8zk\") pod \"maas-api-revoke-sub-a2feff40-sqrgk\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:56.671994 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.671967 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2feff40-sqrgk\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:56.672039 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.671996 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2feff40-sqrgk\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:56.672592 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.672567 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2feff40-sqrgk\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:56.674312 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.674295 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2feff40-sqrgk\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:56.728472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.728440 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj8zk\" (UniqueName: \"kubernetes.io/projected/fbf8d220-bb7a-44f0-ae59-e110ace82b94-kube-api-access-qj8zk\") pod \"maas-api-revoke-sub-a2feff40-sqrgk\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:56.882914 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:56.882829 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:57.249188 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:57.249163 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk"] Nov 03 00:47:57.250196 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:47:57.250168 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf8d220_bb7a_44f0_ae59_e110ace82b94.slice/crio-39d40c1031a30a596dbccbc26af66c7ca12c2917dc47ab3f8fafef45f5d97cec WatchSource:0}: Error finding container 39d40c1031a30a596dbccbc26af66c7ca12c2917dc47ab3f8fafef45f5d97cec: Status 404 returned error can't find the container with id 39d40c1031a30a596dbccbc26af66c7ca12c2917dc47ab3f8fafef45f5d97cec Nov 03 00:47:57.658841 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:57.658757 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" event={"ID":"fbf8d220-bb7a-44f0-ae59-e110ace82b94","Type":"ContainerStarted","Data":"cde52db940f2b6b09cb697e83a0379197f09b6ae2ae2b191f95c52b604bc329b"} Nov 03 00:47:57.658841 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:57.658825 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" event={"ID":"fbf8d220-bb7a-44f0-ae59-e110ace82b94","Type":"ContainerStarted","Data":"39d40c1031a30a596dbccbc26af66c7ca12c2917dc47ab3f8fafef45f5d97cec"} Nov 03 00:47:57.767031 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:57.766983 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" podStartSLOduration=1.766970071 podStartE2EDuration="1.766970071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:57.765603613 +0000 UTC m=+6871.172915454" watchObservedRunningTime="2026-11-03 00:47:57.766970071 +0000 UTC m=+6871.174281926" Nov 03 00:47:58.665447 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:58.665410 5672 generic.go:356] "Generic (PLEG): container finished" podID="fbf8d220-bb7a-44f0-ae59-e110ace82b94" containerID="cde52db940f2b6b09cb697e83a0379197f09b6ae2ae2b191f95c52b604bc329b" exitCode=0 Nov 03 00:47:58.665836 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:58.665456 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" event={"ID":"fbf8d220-bb7a-44f0-ae59-e110ace82b94","Type":"ContainerDied","Data":"cde52db940f2b6b09cb697e83a0379197f09b6ae2ae2b191f95c52b604bc329b"} Nov 03 00:47:59.846586 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:59.846553 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:47:59.902840 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:59.902802 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf8d220-bb7a-44f0-ae59-e110ace82b94-kube-api-access-qj8zk\" (UniqueName: \"kubernetes.io/projected/fbf8d220-bb7a-44f0-ae59-e110ace82b94-kube-api-access-qj8zk\") pod \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " Nov 03 00:47:59.903016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:59.902927 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-xks-ca\") pod \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " Nov 03 00:47:59.903057 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:59.903018 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-openshift-ca\") pod \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\" (UID: \"fbf8d220-bb7a-44f0-ae59-e110ace82b94\") " Nov 03 00:47:59.903660 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:59.903625 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-openshift-ca" pod "fbf8d220-bb7a-44f0-ae59-e110ace82b94" (UID: "fbf8d220-bb7a-44f0-ae59-e110ace82b94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:59.905061 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:59.905035 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf8d220-bb7a-44f0-ae59-e110ace82b94-kube-api-access-qj8zk" pod "fbf8d220-bb7a-44f0-ae59-e110ace82b94" (UID: "fbf8d220-bb7a-44f0-ae59-e110ace82b94"). InnerVolumeSpecName "kube-api-access-qj8zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:59.905886 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:47:59.905860 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-xks-ca" pod "fbf8d220-bb7a-44f0-ae59-e110ace82b94" (UID: "fbf8d220-bb7a-44f0-ae59-e110ace82b94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:48:00.007853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:00.007737 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:00.007853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:00.007779 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf8d220-bb7a-44f0-ae59-e110ace82b94-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:00.007853 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:00.007797 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj8zk\" (UniqueName: \"kubernetes.io/projected/fbf8d220-bb7a-44f0-ae59-e110ace82b94-kube-api-access-qj8zk\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:00.699175 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:00.695306 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" event={"ID":"fbf8d220-bb7a-44f0-ae59-e110ace82b94","Type":"ContainerDied","Data":"39d40c1031a30a596dbccbc26af66c7ca12c2917dc47ab3f8fafef45f5d97cec"} Nov 03 00:48:00.699175 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:00.695374 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39d40c1031a30a596dbccbc26af66c7ca12c2917dc47ab3f8fafef45f5d97cec" Nov 03 00:48:00.699175 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:00.695450 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2feff40-sqrgk" Nov 03 00:48:05.045002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:05.044968 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv"] Nov 03 00:48:05.091399 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:05.091369 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fec9a8b-nq5hv"] Nov 03 00:48:06.711643 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:06.711609 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5569383e-96f7-40ce-9656-65b6876637f8" path="/var/lib/kubelet/pods/5569383e-96f7-40ce-9656-65b6876637f8/volumes" Nov 03 00:48:07.396746 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.396712 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn"] Nov 03 00:48:07.397125 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.397112 5672 cpu_manager.go:414] "containerMap: removing container" podUID="fbf8d220-bb7a-44f0-ae59-e110ace82b94" containerName="revoke-keys" Nov 03 00:48:07.397207 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.397127 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf8d220-bb7a-44f0-ae59-e110ace82b94" containerName="revoke-keys" Nov 03 00:48:07.397246 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.397209 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf8d220-bb7a-44f0-ae59-e110ace82b94" containerName="revoke-keys" Nov 03 00:48:07.401415 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.401396 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:07.471967 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.471938 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8963f07-5vzhn\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:07.472136 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.471986 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxgl\" (UniqueName: \"kubernetes.io/projected/c5ee1138-620e-422e-b094-0a9131546fab-kube-api-access-cqxgl\") pod \"maas-api-revoke-sub-a8963f07-5vzhn\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:07.472136 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.472054 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8963f07-5vzhn\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:07.498515 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.498486 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:48:07.498515 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.498507 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:48:07.498691 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.498547 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:48:07.498691 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.498619 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:48:07.501050 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.501031 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn"] Nov 03 00:48:07.573217 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.573196 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8963f07-5vzhn\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:07.573329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.573237 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxgl\" (UniqueName: \"kubernetes.io/projected/c5ee1138-620e-422e-b094-0a9131546fab-kube-api-access-cqxgl\") pod \"maas-api-revoke-sub-a8963f07-5vzhn\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:07.573329 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.573257 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8963f07-5vzhn\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:07.573875 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.573854 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8963f07-5vzhn\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:07.575489 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.575472 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8963f07-5vzhn\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:07.626008 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.625984 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxgl\" (UniqueName: \"kubernetes.io/projected/c5ee1138-620e-422e-b094-0a9131546fab-kube-api-access-cqxgl\") pod \"maas-api-revoke-sub-a8963f07-5vzhn\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:07.714183 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:07.714102 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:08.084117 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:08.084092 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn"] Nov 03 00:48:08.085720 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:48:08.085683 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ee1138_620e_422e_b094_0a9131546fab.slice/crio-ed7dd9938ff032dd141ba16654ecf074c02f8e832ca631c2fafaf99652159783 WatchSource:0}: Error finding container ed7dd9938ff032dd141ba16654ecf074c02f8e832ca631c2fafaf99652159783: Status 404 returned error can't find the container with id ed7dd9938ff032dd141ba16654ecf074c02f8e832ca631c2fafaf99652159783 Nov 03 00:48:08.741360 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:08.741326 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" event={"ID":"c5ee1138-620e-422e-b094-0a9131546fab","Type":"ContainerStarted","Data":"c103cddf0abe0b80b5b36cb9152914847ed9436cce40c5f1a88a03a3e8bf9304"} Nov 03 00:48:08.741748 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:08.741375 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" event={"ID":"c5ee1138-620e-422e-b094-0a9131546fab","Type":"ContainerStarted","Data":"ed7dd9938ff032dd141ba16654ecf074c02f8e832ca631c2fafaf99652159783"} Nov 03 00:48:08.848854 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:08.848794 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" podStartSLOduration=1.84877662 podStartE2EDuration="1.84877662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:08.847785986 +0000 UTC m=+6882.255097815" watchObservedRunningTime="2026-11-03 00:48:08.84877662 +0000 UTC m=+6882.256088451" Nov 03 00:48:09.749114 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:09.749080 5672 generic.go:356] "Generic (PLEG): container finished" podID="c5ee1138-620e-422e-b094-0a9131546fab" containerID="c103cddf0abe0b80b5b36cb9152914847ed9436cce40c5f1a88a03a3e8bf9304" exitCode=0 Nov 03 00:48:09.749536 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:09.749174 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" event={"ID":"c5ee1138-620e-422e-b094-0a9131546fab","Type":"ContainerDied","Data":"c103cddf0abe0b80b5b36cb9152914847ed9436cce40c5f1a88a03a3e8bf9304"} Nov 03 00:48:11.131351 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.131319 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:11.211584 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.211554 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ee1138-620e-422e-b094-0a9131546fab-kube-api-access-cqxgl\" (UniqueName: \"kubernetes.io/projected/c5ee1138-620e-422e-b094-0a9131546fab-kube-api-access-cqxgl\") pod \"c5ee1138-620e-422e-b094-0a9131546fab\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " Nov 03 00:48:11.211784 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.211619 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-openshift-ca\") pod \"c5ee1138-620e-422e-b094-0a9131546fab\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " Nov 03 00:48:11.211784 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.211659 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-xks-ca\") pod \"c5ee1138-620e-422e-b094-0a9131546fab\" (UID: \"c5ee1138-620e-422e-b094-0a9131546fab\") " Nov 03 00:48:11.212018 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.211985 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-openshift-ca" pod "c5ee1138-620e-422e-b094-0a9131546fab" (UID: "c5ee1138-620e-422e-b094-0a9131546fab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:48:11.213748 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.213723 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-xks-ca" pod "c5ee1138-620e-422e-b094-0a9131546fab" (UID: "c5ee1138-620e-422e-b094-0a9131546fab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:48:11.213748 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.213740 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ee1138-620e-422e-b094-0a9131546fab-kube-api-access-cqxgl" pod "c5ee1138-620e-422e-b094-0a9131546fab" (UID: "c5ee1138-620e-422e-b094-0a9131546fab"). InnerVolumeSpecName "kube-api-access-cqxgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:48:11.313038 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.312962 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqxgl\" (UniqueName: \"kubernetes.io/projected/c5ee1138-620e-422e-b094-0a9131546fab-kube-api-access-cqxgl\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:11.313038 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.312991 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:11.313038 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.313001 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ee1138-620e-422e-b094-0a9131546fab-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:11.784204 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.784123 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" event={"ID":"c5ee1138-620e-422e-b094-0a9131546fab","Type":"ContainerDied","Data":"ed7dd9938ff032dd141ba16654ecf074c02f8e832ca631c2fafaf99652159783"} Nov 03 00:48:11.793174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.784457 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed7dd9938ff032dd141ba16654ecf074c02f8e832ca631c2fafaf99652159783" Nov 03 00:48:11.793174 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:11.784541 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8963f07-5vzhn" Nov 03 00:48:16.044499 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:16.044467 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx"] Nov 03 00:48:16.090737 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:16.090695 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad5ca24-f7rdx"] Nov 03 00:48:16.711237 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:16.711206 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1459a01-67fa-4456-b3cc-933634508baa" path="/var/lib/kubelet/pods/a1459a01-67fa-4456-b3cc-933634508baa/volumes" Nov 03 00:48:18.234277 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.234240 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp"] Nov 03 00:48:18.234817 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.234686 5672 cpu_manager.go:414] "containerMap: removing container" podUID="c5ee1138-620e-422e-b094-0a9131546fab" containerName="revoke-keys" Nov 03 00:48:18.234817 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.234701 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ee1138-620e-422e-b094-0a9131546fab" containerName="revoke-keys" Nov 03 00:48:18.234817 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.234779 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ee1138-620e-422e-b094-0a9131546fab" containerName="revoke-keys" Nov 03 00:48:18.239002 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.238980 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:18.275454 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.275425 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee133a8-2735-49e7-9850-c4da68868889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0015ed60-vbfcp\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:18.275563 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.275490 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee133a8-2735-49e7-9850-c4da68868889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0015ed60-vbfcp\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:18.275618 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.275581 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkc5x\" (UniqueName: \"kubernetes.io/projected/cee133a8-2735-49e7-9850-c4da68868889-kube-api-access-pkc5x\") pod \"maas-api-revoke-sub-0015ed60-vbfcp\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:18.290652 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.290623 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp"] Nov 03 00:48:18.334352 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.334313 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:48:18.334525 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.334406 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:48:18.334525 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.334406 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:48:18.334525 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.334431 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:48:18.376889 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.376858 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkc5x\" (UniqueName: \"kubernetes.io/projected/cee133a8-2735-49e7-9850-c4da68868889-kube-api-access-pkc5x\") pod \"maas-api-revoke-sub-0015ed60-vbfcp\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:18.377054 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.376927 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee133a8-2735-49e7-9850-c4da68868889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0015ed60-vbfcp\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:18.377054 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.376993 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee133a8-2735-49e7-9850-c4da68868889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0015ed60-vbfcp\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:18.377646 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.377624 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee133a8-2735-49e7-9850-c4da68868889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0015ed60-vbfcp\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:18.379311 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.379288 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee133a8-2735-49e7-9850-c4da68868889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0015ed60-vbfcp\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:18.431343 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.431319 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkc5x\" (UniqueName: \"kubernetes.io/projected/cee133a8-2735-49e7-9850-c4da68868889-kube-api-access-pkc5x\") pod \"maas-api-revoke-sub-0015ed60-vbfcp\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:18.551676 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.551585 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:18.921114 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:18.920857 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp"] Nov 03 00:48:18.926801 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:48:18.926772 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee133a8_2735_49e7_9850_c4da68868889.slice/crio-2c1f99b3b1afc66aaa9477e7b5919dffda6b7e2d41e5d957318aa8fbca44ae73 WatchSource:0}: Error finding container 2c1f99b3b1afc66aaa9477e7b5919dffda6b7e2d41e5d957318aa8fbca44ae73: Status 404 returned error can't find the container with id 2c1f99b3b1afc66aaa9477e7b5919dffda6b7e2d41e5d957318aa8fbca44ae73 Nov 03 00:48:19.835036 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:19.835001 5672 generic.go:356] "Generic (PLEG): container finished" podID="cee133a8-2735-49e7-9850-c4da68868889" containerID="ba9726b26a1b5b320821b379e3d4cd665040b75377b31561eff274e669bb0717" exitCode=0 Nov 03 00:48:19.835489 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:19.835088 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" event={"ID":"cee133a8-2735-49e7-9850-c4da68868889","Type":"ContainerDied","Data":"ba9726b26a1b5b320821b379e3d4cd665040b75377b31561eff274e669bb0717"} Nov 03 00:48:19.835489 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:19.835162 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" event={"ID":"cee133a8-2735-49e7-9850-c4da68868889","Type":"ContainerStarted","Data":"2c1f99b3b1afc66aaa9477e7b5919dffda6b7e2d41e5d957318aa8fbca44ae73"} Nov 03 00:48:20.999163 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:20.999130 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:21.100925 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.100845 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee133a8-2735-49e7-9850-c4da68868889-kube-api-access-pkc5x\" (UniqueName: \"kubernetes.io/projected/cee133a8-2735-49e7-9850-c4da68868889-kube-api-access-pkc5x\") pod \"cee133a8-2735-49e7-9850-c4da68868889\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " Nov 03 00:48:21.101094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.100931 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee133a8-2735-49e7-9850-c4da68868889-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee133a8-2735-49e7-9850-c4da68868889-maas-api-openshift-ca\") pod \"cee133a8-2735-49e7-9850-c4da68868889\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " Nov 03 00:48:21.101094 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.101000 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee133a8-2735-49e7-9850-c4da68868889-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee133a8-2735-49e7-9850-c4da68868889-maas-api-xks-ca\") pod \"cee133a8-2735-49e7-9850-c4da68868889\" (UID: \"cee133a8-2735-49e7-9850-c4da68868889\") " Nov 03 00:48:21.101420 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.101391 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee133a8-2735-49e7-9850-c4da68868889-maas-api-openshift-ca" pod "cee133a8-2735-49e7-9850-c4da68868889" (UID: "cee133a8-2735-49e7-9850-c4da68868889"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:48:21.103503 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.103477 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee133a8-2735-49e7-9850-c4da68868889-maas-api-xks-ca" pod "cee133a8-2735-49e7-9850-c4da68868889" (UID: "cee133a8-2735-49e7-9850-c4da68868889"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:48:21.103695 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.103670 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee133a8-2735-49e7-9850-c4da68868889-kube-api-access-pkc5x" pod "cee133a8-2735-49e7-9850-c4da68868889" (UID: "cee133a8-2735-49e7-9850-c4da68868889"). InnerVolumeSpecName "kube-api-access-pkc5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:48:21.202301 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.202267 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee133a8-2735-49e7-9850-c4da68868889-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:21.202301 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.202293 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkc5x\" (UniqueName: \"kubernetes.io/projected/cee133a8-2735-49e7-9850-c4da68868889-kube-api-access-pkc5x\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:21.202301 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.202303 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee133a8-2735-49e7-9850-c4da68868889-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:21.870250 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.870214 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" event={"ID":"cee133a8-2735-49e7-9850-c4da68868889","Type":"ContainerDied","Data":"2c1f99b3b1afc66aaa9477e7b5919dffda6b7e2d41e5d957318aa8fbca44ae73"} Nov 03 00:48:21.870537 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.870517 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c1f99b3b1afc66aaa9477e7b5919dffda6b7e2d41e5d957318aa8fbca44ae73" Nov 03 00:48:21.870721 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:21.870706 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0015ed60-vbfcp" Nov 03 00:48:22.734172 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:22.733250 5672 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2nshh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:48:22.734172 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:22.733317 5672 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2nshh" podUID="b303eca0-a0b3-48c4-928e-23f3266ae2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:48:26.018965 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:48:26.018924 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:48:29.060234 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.060201 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp"] Nov 03 00:48:29.060636 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.060584 5672 cpu_manager.go:414] "containerMap: removing container" podUID="cee133a8-2735-49e7-9850-c4da68868889" containerName="revoke-keys" Nov 03 00:48:29.060636 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.060595 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee133a8-2735-49e7-9850-c4da68868889" containerName="revoke-keys" Nov 03 00:48:29.060710 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.060651 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee133a8-2735-49e7-9850-c4da68868889" containerName="revoke-keys" Nov 03 00:48:29.063989 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.063967 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:29.114108 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.114086 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:48:29.160395 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.160361 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:48:29.160395 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.160367 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:48:29.160395 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.160388 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-97vpc\"" Nov 03 00:48:29.162847 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.162828 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp"] Nov 03 00:48:29.177775 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.177754 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db737251-6b7cp\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:29.177830 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.177797 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44z8p\" (UniqueName: \"kubernetes.io/projected/6137e712-f449-4ebf-8608-5c283a0cb582-kube-api-access-44z8p\") pod \"maas-api-revoke-sub-db737251-6b7cp\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:29.177879 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.177862 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db737251-6b7cp\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:29.278715 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.278659 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44z8p\" (UniqueName: \"kubernetes.io/projected/6137e712-f449-4ebf-8608-5c283a0cb582-kube-api-access-44z8p\") pod \"maas-api-revoke-sub-db737251-6b7cp\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:29.278902 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.278727 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db737251-6b7cp\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:29.279015 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.278912 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db737251-6b7cp\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:29.279462 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.279437 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db737251-6b7cp\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:29.281424 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.281403 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db737251-6b7cp\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:29.331945 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.331885 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44z8p\" (UniqueName: \"kubernetes.io/projected/6137e712-f449-4ebf-8608-5c283a0cb582-kube-api-access-44z8p\") pod \"maas-api-revoke-sub-db737251-6b7cp\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:29.376847 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.376825 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:29.742433 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.742404 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp"] Nov 03 00:48:29.744094 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:48:29.744069 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6137e712_f449_4ebf_8608_5c283a0cb582.slice/crio-6dc95ee6569ede1b295cf1e47dbad664f02016cd1b95c2f018d8a429f4f014af WatchSource:0}: Error finding container 6dc95ee6569ede1b295cf1e47dbad664f02016cd1b95c2f018d8a429f4f014af: Status 404 returned error can't find the container with id 6dc95ee6569ede1b295cf1e47dbad664f02016cd1b95c2f018d8a429f4f014af Nov 03 00:48:29.916105 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:29.916072 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" event={"ID":"6137e712-f449-4ebf-8608-5c283a0cb582","Type":"ContainerStarted","Data":"6dc95ee6569ede1b295cf1e47dbad664f02016cd1b95c2f018d8a429f4f014af"} Nov 03 00:48:30.922803 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:30.922765 5672 generic.go:356] "Generic (PLEG): container finished" podID="6137e712-f449-4ebf-8608-5c283a0cb582" containerID="ed903f3fda39abd44d46efbf8fce9d966fb754332696799b411598c00e1d57ac" exitCode=0 Nov 03 00:48:30.923204 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:30.922812 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" event={"ID":"6137e712-f449-4ebf-8608-5c283a0cb582","Type":"ContainerDied","Data":"ed903f3fda39abd44d46efbf8fce9d966fb754332696799b411598c00e1d57ac"} Nov 03 00:48:32.384129 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.384100 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:32.414497 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.414428 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-openshift-ca\") pod \"6137e712-f449-4ebf-8608-5c283a0cb582\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " Nov 03 00:48:32.415069 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.415045 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-xks-ca\") pod \"6137e712-f449-4ebf-8608-5c283a0cb582\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " Nov 03 00:48:32.415757 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.415733 5672 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6137e712-f449-4ebf-8608-5c283a0cb582-kube-api-access-44z8p\" (UniqueName: \"kubernetes.io/projected/6137e712-f449-4ebf-8608-5c283a0cb582-kube-api-access-44z8p\") pod \"6137e712-f449-4ebf-8608-5c283a0cb582\" (UID: \"6137e712-f449-4ebf-8608-5c283a0cb582\") " Nov 03 00:48:32.416183 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.415358 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-openshift-ca" pod "6137e712-f449-4ebf-8608-5c283a0cb582" (UID: "6137e712-f449-4ebf-8608-5c283a0cb582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:48:32.416595 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.416574 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-openshift-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:32.420289 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.420257 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-xks-ca" pod "6137e712-f449-4ebf-8608-5c283a0cb582" (UID: "6137e712-f449-4ebf-8608-5c283a0cb582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:48:32.424201 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.421553 5672 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6137e712-f449-4ebf-8608-5c283a0cb582-kube-api-access-44z8p" pod "6137e712-f449-4ebf-8608-5c283a0cb582" (UID: "6137e712-f449-4ebf-8608-5c283a0cb582"). InnerVolumeSpecName "kube-api-access-44z8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:48:32.517186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.517094 5672 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44z8p\" (UniqueName: \"kubernetes.io/projected/6137e712-f449-4ebf-8608-5c283a0cb582-kube-api-access-44z8p\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:32.517186 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.517135 5672 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6137e712-f449-4ebf-8608-5c283a0cb582-maas-api-xks-ca\") on node \"ip-10-0-137-4.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:32.985227 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.984900 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" event={"ID":"6137e712-f449-4ebf-8608-5c283a0cb582","Type":"ContainerDied","Data":"6dc95ee6569ede1b295cf1e47dbad664f02016cd1b95c2f018d8a429f4f014af"} Nov 03 00:48:32.985227 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.984966 5672 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dc95ee6569ede1b295cf1e47dbad664f02016cd1b95c2f018d8a429f4f014af" Nov 03 00:48:32.985227 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:32.985038 5672 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db737251-6b7cp" Nov 03 00:48:41.628793 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:41.628754 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6b47m"] Nov 03 00:48:50.049047 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:50.049009 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8"] Nov 03 00:48:50.104507 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:50.104444 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32bb75e4-x4zt8"] Nov 03 00:48:50.711520 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:48:50.711485 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3c51b6-6520-4fc8-a630-2661eabcbeb7" path="/var/lib/kubelet/pods/ee3c51b6-6520-4fc8-a630-2661eabcbeb7/volumes" Nov 03 00:48:56.018216 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:48:56.018171 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:49:13.439762 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:13.439676 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2f5hv_c1428a43-3796-41a6-b8e1-79c8062885b3/manager/0.log" Nov 03 00:49:14.214267 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:14.214239 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-gt9zc_db028e9f-d723-4a0c-998e-e1712a86b183/manager/0.log" Nov 03 00:49:15.132411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:15.132385 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fg848_30e01866-fe73-471f-8e2c-f126cb04b8b8/server/0.log" Nov 03 00:49:15.639270 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:15.639232 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-knpb6_eeb11765-1103-4234-aeef-55ecae7338dd/proxy/0.log" Nov 03 00:49:17.218292 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:17.218266 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0015ed60-vbfcp_cee133a8-2735-49e7-9850-c4da68868889/revoke-keys/0.log" Nov 03 00:49:17.437671 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:17.437639 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0a896428-9phvh_5a6dc5fd-4368-45fd-ae37-3b07f3cf2c7c/revoke-keys/0.log" Nov 03 00:49:17.665045 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:17.664964 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-150516ef-pdh6p_108292e2-a977-4b33-a41b-28ebecedaef7/revoke-keys/0.log" Nov 03 00:49:17.884700 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:17.884672 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-193a5b22-6257c_a596c93a-ab6b-4816-863c-a2a43f4509ff/revoke-keys/0.log" Nov 03 00:49:18.096444 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:18.096410 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-606b245a-25c4v_4ef68fda-9cb4-44b9-9660-da193343b1f4/revoke-keys/0.log" Nov 03 00:49:18.325720 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:18.325688 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-635820af-dx2tl_5e0059da-2321-4ed9-91f3-11af0d82efa7/revoke-keys/0.log" Nov 03 00:49:18.542882 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:18.542852 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-685a8fcb-sk5bh_194aec71-fa21-4c8f-b595-5fff61d68732/revoke-keys/0.log" Nov 03 00:49:18.757109 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:18.757084 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8a9bfad0-4z7jw_738b33f6-0402-4204-8d80-bd7cdc0ab763/revoke-keys/0.log" Nov 03 00:49:18.969346 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:18.969318 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a0955be3-tbr56_fc71ebed-03c4-4e8f-809a-527518a10a51/revoke-keys/0.log" Nov 03 00:49:19.196843 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:19.196813 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a2feff40-sqrgk_fbf8d220-bb7a-44f0-ae59-e110ace82b94/revoke-keys/0.log" Nov 03 00:49:19.424950 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:19.424859 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a4e0954b-m48wc_55563851-8813-4b9d-8633-a5e53f8a8243/revoke-keys/0.log" Nov 03 00:49:19.643322 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:19.643298 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a8963f07-5vzhn_c5ee1138-620e-422e-b094-0a9131546fab/revoke-keys/0.log" Nov 03 00:49:19.880049 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:19.880024 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d4d0c503-rrcvx_c120bc2e-1ee7-40d5-b462-990eeee794bb/revoke-keys/0.log" Nov 03 00:49:20.097553 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:20.097522 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-db737251-6b7cp_6137e712-f449-4ebf-8608-5c283a0cb582/revoke-keys/0.log" Nov 03 00:49:20.319463 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:20.319434 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e1f8efb8-t297l_55e72fd2-07f7-4d18-8a6c-c54f4eae16af/revoke-keys/0.log" Nov 03 00:49:20.543966 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:20.543933 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f15af6b0-ss9gb_f03e08d3-62f6-4fd0-b0c9-7e62d84c4a70/revoke-keys/0.log" Nov 03 00:49:23.679238 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:23.679213 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-kbd6z_2c50862d-0486-4a19-9f75-4bf4f6d27dc2/kuadrant-console-plugin/0.log" Nov 03 00:49:24.218411 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:24.218378 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7t7hj_e8f30574-3996-4dc8-ba43-46c4c92d2bfe/manager/0.log" Nov 03 00:49:24.461367 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:24.461335 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6b47m_8e9db611-6d4f-4748-a3f4-4e8cfff61634/limitador/0.log" Nov 03 00:49:26.019680 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:49:26.019640 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:49:26.400689 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:26.400601 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cc54b748b-tzzls_d22427e6-5f74-40a2-a89b-68bb7d372b0d/kube-auth-proxy/0.log" Nov 03 00:49:26.841198 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:26.841162 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-2nshh_b303eca0-a0b3-48c4-928e-23f3266ae2bc/istio-proxy/0.log" Nov 03 00:49:27.159412 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:27.159330 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-5w5nb_1e09ce1c-af58-4dd7-af07-389b72297c4b/istio-proxy/0.log" Nov 03 00:49:28.569604 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:28.569577 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-gt4gv_26ee8627-3e4b-486b-9e59-f646fa5f5d65/istio-proxy/0.log" Nov 03 00:49:30.731930 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:30.731899 5672 scope.go:122] "RemoveContainer" containerID="448d7c03466fa7b7d1871ba4276fbf02c51ea99669914caa2bf26513767b4882" Nov 03 00:49:30.740795 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:30.740772 5672 scope.go:122] "RemoveContainer" containerID="de826658fece8e2d635132e141b886338419ccf6b922e341e5ca312fe5919b44" Nov 03 00:49:32.042699 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:32.042660 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v"] Nov 03 00:49:32.088649 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:32.088618 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-606b245a-25c4v"] Nov 03 00:49:32.711350 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:32.711313 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef68fda-9cb4-44b9-9660-da193343b1f4" path="/var/lib/kubelet/pods/4ef68fda-9cb4-44b9-9660-da193343b1f4/volumes" Nov 03 00:49:43.047817 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:43.047781 5672 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw"] Nov 03 00:49:43.093866 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:43.093836 5672 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a9bfad0-4z7jw"] Nov 03 00:49:44.711352 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:44.711318 5672 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="738b33f6-0402-4204-8d80-bd7cdc0ab763" path="/var/lib/kubelet/pods/738b33f6-0402-4204-8d80-bd7cdc0ab763/volumes" Nov 03 00:49:53.338347 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:53.338312 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9b6c4_ad49b548-e79b-43aa-9479-57c1e6d06fdd/global-pull-secret-syncer/0.log" Nov 03 00:49:56.019294 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:49:56.019248 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:49:56.861811 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:56.861784 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2f5hv_c1428a43-3796-41a6-b8e1-79c8062885b3/manager/0.log" Nov 03 00:49:56.988719 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:56.988690 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m2smd_52e9c219-9e54-43c4-9dae-7d60c8e33dfe/konnectivity-agent/0.log" Nov 03 00:49:58.082434 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:58.082396 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-gt9zc_db028e9f-d723-4a0c-998e-e1712a86b183/manager/0.log" Nov 03 00:49:58.178190 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:58.178160 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-4.us-west-2.compute.internal_4e49e94ef0b64e567c61a7cf70cf3f00/haproxy/0.log" Nov 03 00:49:58.411012 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:49:58.410928 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fg848_30e01866-fe73-471f-8e2c-f126cb04b8b8/server/0.log" Nov 03 00:50:00.243114 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:00.243079 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-knpb6_eeb11765-1103-4234-aeef-55ecae7338dd/proxy/0.log" Nov 03 00:50:03.868655 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:03.868606 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2f5hv_c1428a43-3796-41a6-b8e1-79c8062885b3/manager/0.log" Nov 03 00:50:05.102598 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.102560 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-gt9zc_db028e9f-d723-4a0c-998e-e1712a86b183/manager/0.log" Nov 03 00:50:05.438197 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.438105 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fg848_30e01866-fe73-471f-8e2c-f126cb04b8b8/server/0.log" Nov 03 00:50:05.674791 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.674754 5672 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m"] Nov 03 00:50:05.675269 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.675250 5672 cpu_manager.go:414] "containerMap: removing container" podUID="6137e712-f449-4ebf-8608-5c283a0cb582" containerName="revoke-keys" Nov 03 00:50:05.675330 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.675273 5672 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6137e712-f449-4ebf-8608-5c283a0cb582" containerName="revoke-keys" Nov 03 00:50:05.675396 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.675384 5672 memory_manager.go:365] "RemoveStaleState removing state" podUID="6137e712-f449-4ebf-8608-5c283a0cb582" containerName="revoke-keys" Nov 03 00:50:05.678567 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.678548 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.727194 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.727143 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cgjxh\"/\"kube-root-ca.crt\"" Nov 03 00:50:05.773510 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.773463 5672 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cgjxh\"/\"openshift-service-ca.crt\"" Nov 03 00:50:05.773510 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.773465 5672 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cgjxh\"/\"default-dockercfg-hnhxn\"" Nov 03 00:50:05.776504 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.776477 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m"] Nov 03 00:50:05.815443 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.815417 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-lib-modules\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.815631 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.815478 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-proc\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.815631 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.815580 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-sys\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.815792 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.815649 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-podres\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.815792 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.815701 5672 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4c64\" (UniqueName: \"kubernetes.io/projected/da6e8964-0d32-4460-b9c4-f210c80e25a2-kube-api-access-b4c64\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.852375 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.852345 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-kbd6z_2c50862d-0486-4a19-9f75-4bf4f6d27dc2/kuadrant-console-plugin/0.log" Nov 03 00:50:05.916882 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.916854 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-podres\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.917035 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.916904 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4c64\" (UniqueName: \"kubernetes.io/projected/da6e8964-0d32-4460-b9c4-f210c80e25a2-kube-api-access-b4c64\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.917035 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.916933 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-lib-modules\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.917035 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.916984 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-proc\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.917035 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.916995 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-podres\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.917035 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.917021 5672 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-sys\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.917268 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.917125 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-lib-modules\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.917268 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.917137 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-sys\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.917387 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.917364 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da6e8964-0d32-4460-b9c4-f210c80e25a2-proc\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.970849 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.970822 5672 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4c64\" (UniqueName: \"kubernetes.io/projected/da6e8964-0d32-4460-b9c4-f210c80e25a2-kube-api-access-b4c64\") pod \"perf-node-gather-daemonset-8sp5m\" (UID: \"da6e8964-0d32-4460-b9c4-f210c80e25a2\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:05.991808 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:05.991733 5672 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:06.360663 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:06.360583 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7t7hj_e8f30574-3996-4dc8-ba43-46c4c92d2bfe/manager/0.log" Nov 03 00:50:06.364016 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:06.363991 5672 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m"] Nov 03 00:50:06.365832 ip-10-0-137-4 kubenswrapper[5672]: W1103 00:50:06.365804 5672 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podda6e8964_0d32_4460_b9c4_f210c80e25a2.slice/crio-43c639a1be93d415c8943a8c58c52031f511c21f1ec393255eec3a500c0fbd50 WatchSource:0}: Error finding container 43c639a1be93d415c8943a8c58c52031f511c21f1ec393255eec3a500c0fbd50: Status 404 returned error can't find the container with id 43c639a1be93d415c8943a8c58c52031f511c21f1ec393255eec3a500c0fbd50 Nov 03 00:50:06.525518 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:06.525480 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" event={"ID":"da6e8964-0d32-4460-b9c4-f210c80e25a2","Type":"ContainerStarted","Data":"7d4a40af06efdb555d542de78d7557eeb685c3dbdc94b305a8f35d647bb97b7f"} Nov 03 00:50:06.525787 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:06.525759 5672 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" event={"ID":"da6e8964-0d32-4460-b9c4-f210c80e25a2","Type":"ContainerStarted","Data":"43c639a1be93d415c8943a8c58c52031f511c21f1ec393255eec3a500c0fbd50"} Nov 03 00:50:06.526190 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:06.526170 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:06.634538 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:06.634436 5672 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" podStartSLOduration=1.634410739 podStartE2EDuration="1.634410739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:06.632910227 +0000 UTC m=+7000.040222084" watchObservedRunningTime="2026-11-03 00:50:06.634410739 +0000 UTC m=+7000.041722575" Nov 03 00:50:06.722903 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:06.722873 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6b47m_8e9db611-6d4f-4748-a3f4-4e8cfff61634/limitador/0.log" Nov 03 00:50:07.247227 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:07.247194 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-knpb6_eeb11765-1103-4234-aeef-55ecae7338dd/proxy/0.log" Nov 03 00:50:10.382191 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:10.382133 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2f5hv_c1428a43-3796-41a6-b8e1-79c8062885b3/manager/0.log" Nov 03 00:50:11.710123 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:11.710087 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-gt9zc_db028e9f-d723-4a0c-998e-e1712a86b183/manager/0.log" Nov 03 00:50:12.119719 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:12.119694 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-fg848_30e01866-fe73-471f-8e2c-f126cb04b8b8/server/0.log" Nov 03 00:50:12.542284 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:12.542257 5672 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-8sp5m" Nov 03 00:50:12.965302 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:12.965212 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qvnbn_fe3683b5-cfe1-408e-be5b-5a81c36304c4/kube-state-metrics/0.log" Nov 03 00:50:13.366117 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:13.366079 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qvnbn_fe3683b5-cfe1-408e-be5b-5a81c36304c4/kube-rbac-proxy-main/0.log" Nov 03 00:50:13.763852 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:13.763817 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-knpb6_eeb11765-1103-4234-aeef-55ecae7338dd/proxy/0.log" Nov 03 00:50:13.773085 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:13.773059 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qvnbn_fe3683b5-cfe1-408e-be5b-5a81c36304c4/kube-rbac-proxy-self/0.log" Nov 03 00:50:14.610475 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:14.610446 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-zhjrz_e26c39e8-2ffc-4179-8498-76167f635e9a/monitoring-plugin/0.log" Nov 03 00:50:16.264464 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:16.264433 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4vrnv_41883b9a-e520-4e4d-8753-4512e53ff141/node-exporter/0.log" Nov 03 00:50:16.675583 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:16.675509 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4vrnv_41883b9a-e520-4e4d-8753-4512e53ff141/kube-rbac-proxy/0.log" Nov 03 00:50:17.083164 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:17.083118 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4vrnv_41883b9a-e520-4e4d-8753-4512e53ff141/init-textfile/0.log" Nov 03 00:50:18.865303 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:18.865267 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cc54b748b-tzzls_d22427e6-5f74-40a2-a89b-68bb7d372b0d/kube-auth-proxy/0.log" Nov 03 00:50:18.971284 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:18.971250 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lh4jx_7d17af01-21f0-48b5-b6b3-834a2e2c7852/kube-rbac-proxy-main/0.log" Nov 03 00:50:19.292710 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:19.292679 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-2nshh_b303eca0-a0b3-48c4-928e-23f3266ae2bc/istio-proxy/0.log" Nov 03 00:50:19.377077 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:19.377049 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lh4jx_7d17af01-21f0-48b5-b6b3-834a2e2c7852/kube-rbac-proxy-self/0.log" Nov 03 00:50:19.505271 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:19.505242 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-5w5nb_1e09ce1c-af58-4dd7-af07-389b72297c4b/istio-proxy/0.log" Nov 03 00:50:19.783926 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:19.783898 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lh4jx_7d17af01-21f0-48b5-b6b3-834a2e2c7852/openshift-state-metrics/0.log" Nov 03 00:50:20.204001 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:20.203932 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f5f0355-698a-4964-a644-68568dbbfe33/prometheus/0.log" Nov 03 00:50:20.565472 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:20.565444 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-gt4gv_26ee8627-3e4b-486b-9e59-f646fa5f5d65/istio-proxy/0.log" Nov 03 00:50:20.610729 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:20.610709 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f5f0355-698a-4964-a644-68568dbbfe33/config-reloader/0.log" Nov 03 00:50:21.018112 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:21.018058 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f5f0355-698a-4964-a644-68568dbbfe33/thanos-sidecar/0.log" Nov 03 00:50:21.426553 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:21.426478 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f5f0355-698a-4964-a644-68568dbbfe33/kube-rbac-proxy-web/0.log" Nov 03 00:50:21.833431 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:21.833400 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f5f0355-698a-4964-a644-68568dbbfe33/kube-rbac-proxy/0.log" Nov 03 00:50:22.240401 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:22.240371 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f5f0355-698a-4964-a644-68568dbbfe33/kube-rbac-proxy-thanos/0.log" Nov 03 00:50:22.646797 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:22.646727 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9f5f0355-698a-4964-a644-68568dbbfe33/init-config-reloader/0.log" Nov 03 00:50:25.577791 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:25.577759 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b8df694dc-d54td_4d158803-71d8-447b-8e4d-e73287682505/thanos-query/0.log" Nov 03 00:50:26.019522 ip-10-0-137-4 kubenswrapper[5672]: E1103 00:50:26.019489 5672 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j2jn2" containerName="extractor" Nov 03 00:50:26.020937 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:26.020915 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b8df694dc-d54td_4d158803-71d8-447b-8e4d-e73287682505/kube-rbac-proxy-web/0.log" Nov 03 00:50:26.429548 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:26.429476 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b8df694dc-d54td_4d158803-71d8-447b-8e4d-e73287682505/kube-rbac-proxy/0.log" Nov 03 00:50:26.835958 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:26.835931 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b8df694dc-d54td_4d158803-71d8-447b-8e4d-e73287682505/prom-label-proxy/0.log" Nov 03 00:50:27.245301 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:27.245270 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b8df694dc-d54td_4d158803-71d8-447b-8e4d-e73287682505/kube-rbac-proxy-rules/0.log" Nov 03 00:50:27.722529 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:27.722502 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b8df694dc-d54td_4d158803-71d8-447b-8e4d-e73287682505/kube-rbac-proxy-metrics/0.log" Nov 03 00:50:28.108527 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:28.108453 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-kbd6z_2c50862d-0486-4a19-9f75-4bf4f6d27dc2/kuadrant-console-plugin/0.log" Nov 03 00:50:28.401118 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:28.401027 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7t7hj_e8f30574-3996-4dc8-ba43-46c4c92d2bfe/manager/0.log" Nov 03 00:50:28.528349 ip-10-0-137-4 kubenswrapper[5672]: I1103 00:50:28.528323 5672 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6b47m_8e9db611-6d4f-4748-a3f4-4e8cfff61634/limitador/0.log"